OS Boot Failure Due to Bcache Registration Timeout

Symptom

The OS fails to boot due to bcache module registration timeout.

Kunpeng servers are used to run Ceph. Each server is configured with 20 HDDs and 4 SSDs. Using the bcache scheme, each SSD is divided into 5 partitions serving as cache partitions for 5 HDDs, resulting in 20 bcache partitions running 20 OSDs. During testing, rebooting the server causes boot failure, preventing access to the system.

Key Process and Cause Analysis

Analyze the time distribution during bcache registration through manual timestamp tracking. The results show that registering front-end partitions is completed quickly. However, registering cache partitions takes over 10 seconds per partition. A total of 20 cache partitions need to be registered, while the default udev device startup timeout is 180 seconds. As a result, some devices cannot complete registration within the timeout limit and are terminated. Further analysis reveals that the bch_tree_check function consumed most of the time during cache partition registration. This function validates keys in the cache B-tree to detect invalid or corrupted keys, ensuring data consistency. When there are a large number of keys, this validation process becomes slow.

Mar 05 12:00:15 hwceph1 systemd-udevd[9474]: worker [9476] /devices/pci0000:00/0000:00:08.0/0000:03:00.0/host4/target4:0:4/4:0:4:0/block/sdd/sdd1 timeout; kill it
Mar 05 12:00:15 hwceph1 systemd-udevd[9474]: seq 6970
/devices/pci0000:00/0000:00:08.0/0000:03:00.0/host4/target4:0:6/4:0:6:0/block/sdf/sdf1 timeout; kill it
Mar 05 12:00:15 hwceph1 systemd-udevd[9474]: seq 6984 '/devices/pci0000:00/0000:00:08.0/0000:03:00.0/host4/target4:0:6/4:0:6:0/block/sdf/sdf1' killed
Mar 05 12:00:15 hwceph1 systemd-udevd[9474]: worker [9500] /devices/pci0000:00/0000:00:08.0/0000:03:00.0/host4/target4:0:25/4:0:25:0/block/sdx/sdx12 timeout; kill it

Conclusion and Solution

• Accelerate bcache registration (preliminary verification passed).

  Accelerate the execution of bch_btree_check. An existing patch (https://patchwork.kernel.org/patch/11383771/) enables multi-threaded parallel validation.

• Increase the udev device management timeout limit (verification passed).

  Increase the timeout limit for udev device management events to prevent bcache-register from being terminated. This allows bcache partitions to register successfully. Add udev.event_timeout=300 to the boot parameters. The timeout value can be adjusted as needed.