Heartbeat Status of a Server in the Cluster Is Down During OBS Testing

Symptom

Three Kunpeng servers are used to deploy Ceph. During the Ceph OBS test on the Kunpeng 920 platform, the heartbeat status of one server in the cluster is consistently down, while the other two servers function normally.

Key Process and Cause Analysis

Network, router, and firewall issues have been ruled out.

osd.0 sends beacons at an interval of 300s, which is configured by the OSD parameter osd_beacon_report_interval. On the monitor side, if no beacon is received from osd.0 within 300s, osd.0 is marked down. Immediately after osd.0 is marked down, a beacon from osd.0 is received. It is recommended to check whether the monitor parameter mon_osd_report_timeout is set to a value less than 300s.

Conclusion and Solution

Change the value of the mon_osd_report_timeout parameter of the monitor to a value greater than 300s.