Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Optimizing Nginx

Purpose

Modify the Nginx configuration file to improve the Nginx service performance.

Procedure

  1. Modify the worker_processes and worker_cpu_affinity configurations in the nginx.conf file based on the number of tested service cores.

    Table 1 describes the configurations.

    Tuning guideline: On the premise that service cores are fully occupied, use as few interrupt cores as possible and use them together with the interrupt script described in Binding NIC Interrupts to Cores.

  2. Bind the Nginx master process to cores.
    taskset -c N /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
    • N indicates the numbers of the cores. For example, 0–2 indicates cores 0 to 2.
    • Binding policy: Bind the Nginx master process to the node or CPU where the service cores reside.
  3. Configure reuseport to avoid socket bottleneck.
  4. When configuring the server, add the reuseport configuration to invoke the SO_USEPORT feature of the kernel and create sufficient sockets to avoid resource contention due to insufficient sockets.
    server {          listen       10000 reuseport;          server_name  localhost;

Nginx.conf Example

In the following example, Nginx is used as a load balancer to process HTTP short connections.

user  root; 
worker_processes 2; 
worker_cpu_affinity 
1 
10; 
error_log /dev/null; 
worker_rlimit_nofile 102400; 
events { 
    worker_connections  102400; 
    multi_accept on; 
    use epoll; 
    accept_mutex off; 
} 
http { 
    include       mime.types; 
    default_type  application/octet-stream; 
    access_log /dev/null; 
    sendfile        on; 
    tcp_nopush     on; 
    tcp_nodelay     on; 
    sendfile_max_chunk 512k; 
    keepalive_timeout  65; 
    keepalive_requests 2000; 
    client_header_buffer_size 4k; 
    large_client_header_buffers 4 32k; 
    server_names_hash_bucket_size 128; 
    client_max_body_size 100m; 
    open_file_cache max=102400 inactive=40s; 
    open_file_cache_valid 50s; 
    open_file_cache_min_uses 1; 
    open_file_cache_errors on; 
   upstream test{ 
       server 192.168.1.75:10000  ; 
       server 192.168.2.75:11000  ; 
       server 192.168.1.72:10000 ; 
       server 192.168.2.72:11000 ; 
       keepalive 300; 
        } 
   server { 
        listen       10000 reuseport; 
        server_name  localhost; 
        access_log off; 
        location / { 
            root   html; 
            index  index.html index.htm; 
                   proxy_pass http://test; 
                proxy_connect_timeout 75; 
                proxy_read_timeout 300; 
                proxy_send_timeout 300; 
                proxy_buffer_size 4k; 
                proxy_buffers   4 32k; 
                proxy_busy_buffers_size 64k; 
                proxy_temp_file_write_size 64k; 
                proxy_http_version 1.1; 
                proxy_set_header Connection ""; 
                proxy_headers_hash_max_size 51200; 
                proxy_headers_hash_bucket_size 6400; 
        } 
        error_page   500 502 503 504  /50x.html; 
        location = /50x.html { 
            root   html; 
        } 
    } 
    # HTTPS server 
        server { 
         listen       20000 ssl ; 
         server_name  localhost; 
         ssl_certificate      /usr/local/nginx/server_2048.crt; 
         ssl_certificate_key  /usr/local/nginx/server_2048.key; 
         ssl_session_cache    shared:SSL:1m; 
         ssl_session_timeout  5m; 
        ssl_protocols  TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; 
        ssl_ciphers  "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4"; 
         ssl_prefer_server_ciphers  on; 
         location / { 
             root   html; 
             index  index.html index.htm; 
         } 
     } 
}

Nginx configuration is based on blocks, including global configuration (such as user and worker_processes) and module configuration (such as the events module and http module). Each module configuration is divided into blocks. For example, the http module has the server block, and the server has the location block.

Table 1 describes basic Nginx configuration and parameters

Table 1 Basic Nginx configuration and parameters

Module

Parameter

Description

Global

user root;

Specifies the user and user group for running Nginx.

worker_processes 3;

Specifies the number of Nginx processes. It is advised to set the value of this parameter to the total number of CPU cores.

worker_cpu_affinity

1

10

100;

Binds a CPU to each worker process by using worker_cpu_affinity. 1 indicates core 0, 10 indicates core 1, 100 indicates core 2, ..., 100000 indicates core 5. The number of 0s indicates the sequence number of the core.

error_log /dev/null;

Specifies the error log file path. /dev/null indicates that Nginx logs are disabled.

worker_rlimit_nofile 102400;

Specifies the maximum number of file descriptors that can be accessed by an Nginx process, which is limited by the maximum number of files that can be opened by a system process (ulimit -n).

Events

worker_connections 102400;

Specifies the maximum number of connections for each process, which is limited by the maximum number of files that can be opened by a system process. The total number of concurrent connections is the product of worker_processes and worker_connections.

multi_accept on;

Sets whether a process can accept multiple network connections at the same time. If multi_accept is disabled, a worker process of Nginx can accept only one new connection each time.

use epoll;

Sets the working mode to epoll. In addition to epoll, the working mode can be select, poll, kqueue, rtsig, or /dev/poll.

accept_mutex on;

Enables accept_mutex to prevent context switching and try_lock lock overhead caused by worker resource contention.

HTTP

include mime.types;

Specifies the table of mapping between file name extensions and file types.

default_type application/octet-stream;

Specifies the default file type.

access_log /dev/null;

Specifies the Nginx access log path. /dev/null indicates that the access log is disabled.

sendfile on;

Enables the efficient file transfer mode to obtain files across the user mode. The .gzip compressor needs to be executed in user mode. Therefore, it cannot coexist with sendfile.

tcp_nopush on;

Enables the function of preventing network congestion. The first data packet that returns the data is sent only when it carries the block data obtained from sendfile.

tcp_nodelay on;

Enables the function of preventing network congestion.

sendfile_max_chunk 512k;

Specifies the maximum size of the sendfile file to prevent oversized files from occupying the entire worker process. There is no limit by default.

keepalive_timeout 65;

Specifies the timeout period for keeping the client connections alive.

keepalive_requests 2000;

Specifies the maximum number of requests that can be served by a keepalive connection. When the maximum number of requests is reached, the connection is closed.

client_header_buffer_size 4k;

Specifies the size of the buffer that stores client request headers. Set this parameter based on the system page size. Generally, the header size of a request does not exceed 1 KB. However, the system page must be greater than 1 KB. Therefore, set this parameter to the page size. You can run the getconf PAGESIZE command to obtain the page size.

large_client_header_buffers 4 32k;

Specifies the size of the buffer that stores large client request headers. 4 indicates the number of buffers and 32k indicates the buffer size. 4 32k indicates that four buffers of 32 KB are requested. By default, Nginx uses the client_header_buffer_size buffer to read the header value. If the header value is too large, Nginx uses the large_client_header_buffers buffer to read the header value.

server_names_hash_bucket_size 128;

Specifies the size of the hash table that stores server names.

client_max_body_size 100m;

Controls the size of all Nginx request packets to 100 MB.

open_file_cache max=102400 inactive=40s;

max sets the maximum number of elements in the cache. When the cache overflows, delete the least recently used (LRU) element. inactive defines a time period. If the element is not accessed during this period, delete the element from the cache.

open_file_cache_valid 50s;

Specifies the interval for checking the cached valid information. That is, even if the file is frequently accessed, the system checks for change of the file every 50 seconds. If there is a change, the system updates the file.

open_file_cache_min_uses 1;

Specifies the minimum number of times that a file can be used within the period specified by the inactive parameter in open_file_cache. If the number is exceeded, the file change will be dynamically updated in the cache.

open_file_cache_errors on;

Enables the cache for file searching errors.

Reverse proxy server group test (Nginx distributes requests to a server in the group based on the configuration.)

server 192.168.1.75:10000 ;

server 192.168.2.75:11000;

server 192.168.1.73:10000;

server 192.168.2.73:11000;

Configures the IP address and port of the server that processes the requests.

keepalive 300;

Specifies the connection timeout period.

Server

listen 10000 reuseport;

Sets port 10000 as the listening port. You are advised to enable reuseport in HTTP short connection scenarios. The port can be reused in persistent connection scenarios.

server_name localhost;

Specifies localhost as the access mode.

access_log off;

Disables the access log of the virtual host.

Default request

root html;

Specifies the default root directory location of the website on the server. root is the web root directory.

index index.html index.htm;

Specifies the names and sequence of the index files on the home page.

proxy_pass http://test;

Forwards all requests to the server configured in the test server group.

proxy_connect_timeout 75;

Specifies the maximum time allowed for setting up a connection with the backend server.

proxy_read_timeout 300;

Specifies the maximum time allowed for waiting for a response from the backend server after Nginx is connected to the backend server.

proxy_send_timeout 300;

Specifies the maximum time allowed for transferring requests to the backend server. If timeout occurs, Nginx disconnects the connection.

proxy_buffer_size 4k;

Specifies the size of the proxy buffer for storing user header information.

proxy_buffers 432k;

Specifies the number of buffers and the buffer size.

proxy_busy_buffers_size 64k;

Sets larger proxy_buffers if the system load is heavy and the buffer is insufficient.

proxy_temp_file_write_size 64k;

Specifies the size of the temporary cache file.

proxy_http_version 1.1;

Sets the Nginx server to use HTTP/1.1 to provide proxy services.

proxy_set_header Connection "";

Allows some request headers that will be transferred to the proxy server to be redefined and added.

proxy_headers_hash_max_size 51200;

Specifies the maximum capacity of the hash table that stores HTTP packet headers.

proxy_headers_hash_bucket_size 6400;

Specifies the size of the hash table that the Nginx server requests to store HTTP packet headers.

error_page 500 502 503 504 /50x.html;

location = /50x.html {

root html;

Specifies the error page.

SSL encryption

listen 20000 ssl;

Sets port 20000 as the listening port.

server_name localhost;

Specifies localhost as the access mode.

ssl_certificate /usr/local/nginx/server_2048.crt;

ssl_certificate is a public key, which is sent to each client that connects to the server.

ssl_certificate_key /usr/local/nginx/server_2048.key;

The ssl_certificate_key private key is used for decryption. Therefore, its permission must be protected, but the master process of Nginx should be able to read it. The private key and certificate can be stored in the same certificate file. In this mode, only the public key certificate is sent to the client.

ssl_session_cache shared:SSL:1m;

Specifies the type and size of the SSL session cache. shared:SSL:1m indicates that all Nginx worker processes share the SSL session cache. A cache of 1 MB can store about 4000 sessions.

ssl_session_timeout 5m;

Specifies the timeout period (5 minutes) for which the client can reuse the SSL parameters in the session cache.

ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;

Specifies the encryption protocols that can be used.

ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

Chooses an encryption suite.

ssl_prefer_server_ciphers on;

Specifies that the encryption suite of the server is preferentially used instead of that of the client browser to negotiate the encryption algorithm.

location / {

root html;

index index.html index.htm;

}

Specifies the default request.

Specifies the default root directory location of the website on the server. root is the web root directory.

Specifies the names and sequence of the index files on the home page.
Parent topic: Application Tuning