Failed to Perform an I/O Analysis Task

Symptom

An I/O analysis task fails. The error message is "system not support, please open ftrace." Check whether the system supports ftrace and whether ftrace is enabled.

Possible Cause

The OS kernel version does not support ftrace.

Troubleshooting Procedure

Ftrace is a tracing tool built in the Linux kernel and has been added to mainstream kernels since the kernel version 2.6.27. You can use ftrace to debug or analyze events in the kernel. Ftrace provides different tracers for different scenarios, such as tracing kernel function calls, tracing context switching, viewing the duration when interrupts are disabled, tracing the delay in the kernel mode, and tracing performance problems. System developers can use ftrace to trace and debug kernels to find the root causes of the problems in the kernels and rectify the problems. In addition, users who are interested in kernels can use ftrace to observe the activities in kernels and understand the working mechanism of kernels.

Perform the following steps to recompile the kernel:

1. Enable the kernel to support ftrace.
   1. Before using ftrace, compile it into the kernel. The kernel/trace/Makefile file in the kernel source code directory provides the compilation options related to ftrace, as shown in Table 1.

      Table 1 Configuration items related to ftrace

      No.	Configuration Item
      1	CONFIG_FUNCTION_TRACER
      2	CONFIG_FUNCTION_GRAPH_TRACER
      3	CONFIG_CONTEXT_SWITCH_TRACER
      4	CONFIG_NOP_TRACER
      5	CONFIG_SCHED_TRACER
      6	...

      Many configuration items are related to ftrace. Different tracers have different configuration items, and thus have different dependencies. The kernel/trace/Kconfig file in the kernel source code directory describes these dependencies. You can refer to the Makefile and Kconfig files and select the required tracer.

   2. Generally, it is more intuitive to use make menuconfig when configuring the kernel. Take kernel 2.6.33.1 as an example. To compile ftrace into the kernel, select Kernel hacking and Tracers, as shown in Figure 1 and Figure 2.
      Figure 1 Linux Kernel Configuration – Kernel hacking
      
      Figure 2 Linux Kernel Configuration – Tracers
      
   3. The Tracers menu lists the tracers supported by the kernel, as shown in Figure 3. You can select a tracer as required.
      Figure 3 Tracers supported by the kernel
      
      

      On a 32-bit x86 server, do not select Optimize for size (see Figure 5) in the General setup menu (see Figure 4) during compilation. Otherwise, the Kernel Function Graph Tracer option in is not displayed on the screen shown in Figure 3. This is because in the Kconfig file, the CONFIG_FUNCTION_GRAPH_TRACER configuration item has a special dependency condition depends on !X86_32 || !CC_OPTIMIZE_FOR_SIZE for a 32-bit x86 server.

      Figure 4 General setup
      
      Figure 5 Optimize for size
      
   4. Ftrace provides access interfaces for the user mode through debugfs. Therefore, debugfs needs to be compiled into the kernel. To activate the support for debugfs for the kernel, set CONFIG_DEBUG_FS to y in the kernel configuration file (.config). You can also run the make menuconfig command and select Debug Filesystem in the Kernel hacking menu to support debugfs, as shown in Figure 6.
      Figure 6 Debug Filesystem
      
   5. After the configuration is complete, compile and install the new kernel, and then start the new kernel.

      After the support for ftrace is activated for the kernel, the -pg option of the compiler is used during kernel compilation. This option is defined in the kernel/trace/Makefile file, as shown in the following figure.

      ifdef CONFIG_FUNCTION_TRACER 
      ORIG_CFLAGS := $(KBUILD_CFLAGS) 
      KBUILD_CFLAGS = $(subst -pg,,$(ORIG_CFLAGS)) 
      ... 
      endif 
      ...

      Using the -pg option adds a lot of debugging information to the compiled kernel image. Generally, the support for ftrace is activated only in the development and test phase to debug the kernel and fix bugs. The kernel eventually used for release will disable the -pg option and will not be able to use ftrace.

2. Access ftrace through debugfs.
   1. Ftrace provides access interfaces for the user mode through debugfs. After debugfs is activated during kernel configuration, the /sys/kernel/debug directory is created. The debugfs file system is mounted to this directory.
      To mount debugfs to the directory, add the following content to the /etc/fstab file:

      1	debugfs /sys/kernel/debug debugfs defaults 0 0

      You can also mount debugfs to the directory when the OS is running.

      1	mount -t debugfs nodev /sys/kernel/debug

   2. After the support for ftrace is activated for the kernel, the /sys/kernel/debug/tracing directory is created in debugfs. This directory contains the ftrace control and output files, as shown in Figure 7. The files and directories in this directory vary according to the settings of ftrace during kernel compilation.
      Figure 7 Files in the tracing directory
      

Data files of ftrace

There are many files and directories in the /sys/kernel/debug/trace directory. Some of them are shared by various tracers, and some are used by a specific tracer. Generally, the echo command is used to change the values of these data files. You can also use the functions related to file reading and writing in the program to change the values of these files. Table 2 describes only some data files of ftrace. For details about the usage of other data files of ftrace, see the documents in the Documentation/trace directory in the kernel source package and the source files in the kernel/trace directory.

Ftrace Tracers

Ftrace contains multiple tracers to trace different types of information, such as information about process scheduling and interrupt disabling. You can obtain from the available_tracers file the list of trackers supported by the kernel. When compiling the kernel, you can see the options corresponding to the trackers supported by the kernel, as described in Figure 3.

Ftrace also supports other tracers, such as initcall, ksym_tracer, mmiotrace, and sysprof. The ftrace framework allows users to add new tracers. You can refer to the documents in the Documentation/trace directory of the kernel source package and the source files in the kernel/trace directory to understand the usage of other tracers and the method for adding new tracers.