OSD Cannot Be Started After the Server Is Rebooted
Symptom
Environment settings
Hardware |
Kunpeng server (25 x 2.5-inch drive expander chassis, 2 x Kunpeng 920 processor, 32 cores@2.6 GHz) 20 x HDD (2400 GB, SAS 12 Gb/s,10k RPM, 256 MB, 2.5-inch drive bay) 4 x SSD (1920 GB, SATA 6 Gb/s, mixed use, SM883 series, 2.5-inch drive bay) |
|---|---|
OS |
CentOS 7.6 (Kernel 4.14) |
Problem description: After the server is rebooted, osd.36 cannot be started properly.
Key Process and Cause Analysis
- Check the ceph-osd.36.log file. It reveals the following error:
ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-36: (2) No such file or directory
- This indicates that the OSD failed to load bcache during startup, resulting in startup failure.
- Locate the server with the missing bcache. Assume that one SSD is mapped to five HDDs in the current environment. A server with 4 SSDs and 20 HDDs will have 40 bcache devices in total.Run the following command on each of the three servers to check the bcache count:
lsblk | grep bcache | wc -l
The output shows that hwceph3 is missing two bcache devices, as shown below.
- Check the locations of the missing bcache devices using lsblk. The results show that the bcache devices for SSD partition sdx12 and HDD partition sdq1 are missing, as shown below:
- Check the ceph-osd.36.log file. It reveals shutdown logs at 12:58:13 on February 11, as shown below:
- Check /var/log/message. It reveals system boot log entries at 13:09:29. This indicates a server reboot, which triggered the shutdown of osd.36, as shown below:
Based on the above information, osd.36 failed to start during the system boot at 13:09:29 because its associated bcache was missing and could not be loaded.
- Check /var/log/message. It shows "journal entries 13614982-13614983 missing" at 13:11:10. This confirms the bcache loss.
- Download the kernel code and analyze the logs. It reveals that during the recovery process after reboot, the journal buckets need to recover entries 13614982 to 13614995. However, entries 13614982 to 13614983 were lost, so only 12 entries were recovered. Consequently, the system reported "journal entries 13614982-13614983 missing!" Upon detecting this data inconsistency, the system shut down the bcache device for sdq1.
Conclusion and Solution
Add judgment conditions to the log information of the bch_journal_replay function. The open-source code is as follows:
git a/drivers/md/bcache/journal.c b/drivers/md/bcache/journal.c
