"Internal Server Error, HTTP 500" Displayed When Executing the openstack network list Command

Symptom

The following information is displayed when the openstack network list command is executed:

The /var/log/neutron/server.log file contains the following information:

oslo_messaging.rpc.server OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connect

See the following figure:

Key Process and Cause Analysis

OpenStack is always in the scheduling state and "Too many connections" is displayed for database connections. This problem occurs when the number of database connections reaches the limit.

Conclusion and Solution

1. Open the MariaDB configuration file.

   vim /etc/my.cnf.d/openstack.cnf

   Locate max_connections under the [mysqld] field and set max_connections = 4096. The following figure shows an example.

   

2. Change the maximum number of files that can be opened in the system.
   1. Open the file.

      vim /etc/security/limits.conf

   2. Add the following information to the file.

      * soft nofile 65536
      * hard nofile 65536

      See the following figure:

      

   3. Add the following information to the /etc/pam.d/login file:

      session required /lib/security/pam_limits.so

      After a user logs in to the system, Linux invokes the pam_limits.so module to limit the maximum number of resources that can be used by the user, including the maximum number of files that can be opened. The pam_limits.so module reads configurations from the /etc/security/limits.conf file.

   4. Set the maximum number of file handles that can be opened by the system:

      ulimit -SHn 65536

3. Set the maximum number of files that can be opened.
   1. Open the file.

      vim /etc/sysctl.conf

   2. Add the following information to the file.

      # max open files
      fs.file-max = 65536

      

   3. Make the configuration take effect.

      sysctl -p

4. Modify database parameters.
   1. Edit the /usr/lib/systemd/system/mariadb.service file and add the following information under [Service] to change the maximum file descriptor limit of MySQL in Linux:

      LimitNOFILE=65535
      LimitNPROC=65535

   2. Make the configuration take effect.

      systemctl daemon-reload
      systemctl restart  mariadb.service

   3. Change the maximum database connections.
      • Query the maximum number of MySQL connections.

        mysql> show variables like 'max_connections';

      • Query the number of current connections of the server.

        mysql> show global status like 'Max_used_connections';

5. Restart the OpenStack service. Restart all the following OpenStack services (you can use a script):
   1. Restart Neutron.

      systemctl restart neutron-server.service neutron-linuxbridge-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service

   2. Restart Nova.

      systemctl restart openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service

   3. Restart Glance.

      systemctl restart openstack-glance-api.service openstack-glance-registry.service

   4. Restart Cinder.

      systemctl restart openstack-cinder-api.service openstack-cinder-scheduler.service