blktrace

Introduction

blktrace is a user-mode tool that provides detailed information about time consumption on the I/O subsystem. You can analyze whether the I/O scheduling is slow or the hardware response is slow based on the information. The auxiliary tool blkparse reads the original output from blktrace and generates a human-readable I/O operation summary. btt is provided as part of the blktrace software package. It analyzes the blktrace output and displays the amount of time that the blktrace output takes in each I/O stack, making it easier to find bottlenecks on the I/O subsystem.

Installation Method

CentOS is used as an example. You can run the following command to install it:

1

# yum -y install blktrace

How to Use

1. Run the blktrace command to capture the I/O information of a specified device. For example:

   1	# blktrace -w 120 -d /dev/sda

   The parameter following -w specifies the capture time, in seconds. The parameter following -d specifies the device.

   After the command is executed, a series of binary files in the device.blktrace.cpu format are generated.

2. Run the blkparse command to parse the data generated by blktrace. For example:

   1	# blkparse -i sda -d blkparse.out

   The parameter following -i specifies the input file name. The output file name of blktrace is in the device.blktrace.cpu format by default. Therefore, you only need to enter device. The parameter following -d specifies the binary output file.

   This command displays the analysis result on the screen and outputs the binary data of the analysis result to the blkparse.out file.

3. Run the btt command to view the overall I/O status.

   1	# btt -i blkparse.out

   The parameter following -i specifies the input file name.

   

   • Q2Q: time interval for sending multiple requests to the block I/O layer.
   • Q2G: time consumed by I/O request generation, including time for remapping (the I/O request may be remapped to other devices by Device Mapper or Multiple Device (Software RAID)) and time for splitting (the I/O request may be split into multiple physical I/Os because the I/O request is not aligned with the sector boundary or the size of the I/O request is too large).
   • G2I: time consumed by an I/O request to enter the I/O scheduler, including the time for merging (the I/O request may be merged with other physically adjacent I/O requests into one large I/O request).
   • I2D: waiting time of an I/O request in the I/O scheduler.
   • D2C: time consumed by an I/O request on the driver and hardware (the I/O request is submitted by the driver to the hardware, passes through the HBA, cables (such as optical fibers and network cables), switch (SAN or network), and finally arrives at the storage device. The device returns the result after completing the I/O request).
   • Q2C: time consumed by the entire I/O request (Q2I + I2D + D2C = Q2C), which is equivalent to the await value of iostat.

   In normal cases, the D2C proportion is greater than 90%. If the I2D proportion is high, you can adjust the I/O scheduling policy.