Read and Write Test Cases

Case Analysis

Random write: The average CPU usage of random write is 8%, but a large number of I/Os are generated. A large number of I/Os are generated when the HLog, MemStore flush, and Compaction processes are written to HBase. Writing HLog generates a large number of small I/Os. Therefore, this test case is sensitive to I/O performance but insensitive to CPU performance. Generally, there are a large number of small network packets. Therefore, the write performance is improved by enabling the LRO function for a NIC.

Random read: The overall CPU usage of the random read test case is low. The overall I/O is read from the HFile to the block cache at the beginning of the test case. Then a large number of cache hits occur. Because many small packets exist on the network, the LRO function is enabled to aggregate small packets.

Sequential scanning: The CPU usage of sequential scanning test cases is low. The overall I/O is read from the HFile to the blockcache during the initial test case execution, and the subsequent data is read from the blockcache. The process of this test case is the same as that of the random read test case. The only difference is that a segment of consecutive key-value range is read in sequence when a key-value is read. Therefore, the network packet aggregation effect is not obvious during scanning.

Slow Sync Tuning

Symptom

In the high-concurrency put scenario (random write test case), the random write performance is poor due to the synchronous write WAL mechanism. The synchronization latency is high in the log.

Problem Analysis

The synchronous write WAL mechanism ensures that data is written to disks to ensure reliability. As a result, the write WAL latency is high during the write process, severely affecting performance.

Solution

Create a RAID 0 array using disks and set the cache policy of the RAID controller card to RAWBC. Use the cache of the RAID controller card for write back instead of write through, which greatly improves random write performance.

Cache policy configuration for RAID groups:

RWBC -> Read ahead + Write back + Cached IO

NIC Parameter Tuning

In the HBase read/write test scenario, the size of the test record bar is 1 KB. Therefore, enable the LRO function of the NIC and increase the ring buffer size of the NIC. Aggregate small packets on the network to improve network performance.