Reducing Memory Copy

Principles

Based on data flow analysis, the number of memory copy operations is reduced, which reduces the CPU usage and memory bandwidth usage.

Modification Method

Reducing memory copy is based on service logic. The following lists several mechanisms for reducing memory copy:

• Example 1: Use sendfile to replace functions such as send, sendto, and write to send files to the peer end.

  The following two statements send the file to the peer end. Generally, four memory copy operations are performed.

  1 2	ssize_t read (int fd, void *buf, size_t count); ssize_t send (int s, const void * buf, size_t len, int flags);

  • The read function performs memory copy twice. The DMA transfers data to the PageCache of the kernel. The kernel transfers data to the buffer in the application mode.
  • The send function performs memory copy twice. The write function copies the buffer in the application mode to the kernel. The DMA transfers data to the NIC.

    Only two memory copy operations are performed when the following function implementation is used:

    1	ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

    The kernel transfers the file to the cache (first memory copy) through DMA, and then transfers the description information (location and length) of the cache to the TCP/IP protocol stack. The kernel transfers the cache to the NIC (second memory copy) through DMA.

    In addition to code modification, some open-source software already supports this feature. For example, Nginx can use the sendfile on parameter to enable this function.

• Example 2: The shared memory is used to replace the socket/pipe communication for inter-process communication.

  The memory sharing mode allows multiple processes to operate the same memory area. Compared with the socket communication mode, the memory sharing mode requires fewer memory copy operations. Applications can use functions such as shmget to implement inter-process communication.