C/C++ Process Ends Unexpectedly

Fault Locating

The C/C++ process is not executed according to the design requirements and ends unexpectedly. Figure 1 shows how to locate and rectify the fault.

Figure 1 Fault locating for the abnormal end of a C/C++ process

1. Confirm that the C/C++ program ends unexpectedly.
2. Modify the compilation script or enable the debugging macro to recompile a program that can be debugged.
3. Enable the core dump function so that a core dump file can be generated when an exception occurs.
4. Run the program. When the program stops unexpectedly, confirm that a core dump file is generated in the specified path.
5. Use GDB to debug the core dump file and locate the cause.
6. Modify and recompile the code. Then perform a verification.
   • If the problem is resolved, integrate the modification into the code.
   • If the problem persists, generate a core dump file again to locate the fault. If the locating information is insufficient, add the locating information to the code and compile and run the program again.

Case: Program Exception Caused by Memory Overwriting

Symptom

Some software exits unexpectedly on a server.

Fault Locating

1. Run the top command to check whether the related process is ended. If yes, it is an abnormal exit.
2. Add the -g compilation option to makefile to recompile the code.
3. Enable core dump and set a path for storing the generated core dump file.

   1 2	ulimit -c unlimited echo "/home/core.%e.%p.%t" > /proc/sys/kernel/core_pattern

4. Run the program. After the program stops unexpectedly, the core.dsa_sign_multi.xxx.xxx file is generated in the specified path.
5. Use GDB to debug the core dump file. The debugging window is displayed, as shown in the following figure.

   1	gdb dsa_sign_multi core.dsa_sign_multi.xxx.xxx

   

6. Run the info threads command to view the thread information of the process.

   

7. Run the thread command to switch the thread ID to be queried and run the bt command to view the thread stack.

   1 2	thread $ID bt

   

8. Analyze the stack information. It is found that the problem is caused by the calling of the OpenSSL library. However, the OpenSSL used by the system cannot be debugged. As a result, the stack information of OpenSSL cannot be viewed.
9. Recompile a debuggable version of OpenSSL and link the program. Run the program again, generate a new core dump file, debug the file, and view related stack information.

   

10. Check the parameters of the stack that reports the error.

    1 2	info locals f xx

    

    

11. Check the source code implementation.

    

    MD_Update() is the macro alias of EVP_DigestUpdate().

    

12. Find the implementation of the EVP_DigestUpdate() function.

    

    The value of the third parameter of the function must be an unsigned integer.

    The third parameter is (MD_DIGEST_LENGTH/2 – k), and k is calculated based on (st_idx + MD_DIGEST_LENGTH/2). MD_DIGEST_LENGTH is a constant, and st_idx is a variable.

13. Check the code implementation. It is found that st_idx is copied from the global variable state_index. Then, state_index is accumulated and converted to prevent the value from exceeding the value of STATE_SIZE.

    

14. In a normal process, st_idx does not exceed STATE_SIZE. However, in a multi-thread environment, there is a low probability that state_index is used by other threads before being converted. As a result, st_idx exceeds STATE_SIZE, and the result of (MD_DIGEST_LENGTH/2 – k) is negative. After the result is transferred in EVP_DigestUpdate(), it is calculated as an unsigned number. As a result, an out-of-bound access error occurs, generating a core dump.
15. Lock the static variable state_index, recompile the code, and verify that the problem does not occur. Then incorporate the modification into the code. The problem is resolved.