Some Connections Fail During High-Concurrency Data Collection

Symptom

Server cluster information fails to be collected. The error message is as follows:

Probable Causes

Server cluster information collection uses concurrent SSH logins to connect to multiple servers simultaneously. This can cause the number of unauthenticated connections established within a short period to exceed the default sshd threshold. When the concurrency reaches a certain level (for example, 10 or more), the random rejection mechanism of MaxStartups may be triggered, or connections may be fully rejected, causing some collection tasks to fail because their connections are actively terminated.

Handling Procedure

Modify the sshd concurrent connection limit.

1. Log in to the server to be collected and open the sshd_config configuration file.

   vi /etc/ssh/sshd_config

   Locate the MaxStartups setting, as shown in the following figure:

   

2. Press i to enter the insert mode and modify the MaxStartups setting.

   For example:

   MaxStartups 50:30:100

   

   The MaxStartups parameter limits the number of concurrent connections that the SSH daemon waits to authenticate. The format is Maximum_number_of_unauthenticated_connections:Rejection_probability_percentage:Maximum_number_of_connections. The following uses the 50:30:100 value as an example:

   • 0 to 50 connections: The server accepts all new connections.
   • 51 to 100 connections: When the number of unauthenticated connections exceeds 50, the server starts randomly rejecting new connections starting with the 51st connection. The rejection probability starts at 30% and increases linearly as the number of connections increases, reaching 100% when the number of connections reaches 100.
   • More than 100 connections: The server directly drops all new SSH connection requests until the number of active connections falls below 10.

   Set this parameter based on the actual number of concurrent connections. The maximum number of unauthenticated connections must be greater than 0 and less than or equal to the maximum number of connections. The rejection probability percentage must be between 1 and 100.

3. Press Esc, type :wq!, and press Enter to save the file and exit.
4. After setting the parameter, perform a syntax check.

   sshd -t && echo "config OK"

   The following message is displayed when the setting passes the syntax check:

   config OK

5. Gracefully reload the configuration without disconnecting existing connections.

   systemctl reload sshd

6. Retry server cluster information collection.