Ceph OSD Occasionally Receives the SIGHUP Signal
Symptom
Environment settings
Hardware |
Kunpeng server (25 x 2.5-inch drive expander chassis, 2 x Kunpeng 920 processor, 32 cores@2.6 GHz) 20 x HDD (2400 GB, SAS 12 Gb/s,10k RPM, 256 MB, 2.5-inch drive bay) 4 x SSD (1920 GB, SATA 6 Gb/s, mixed use, SM883 series, 2.5-inch drive bay) |
|---|---|
OS |
CentOS 7.6 (Kernel 4.14) |
Problem description: The Ceph OSD occasionally receives the SIGHUP signal, and the Ceph OSD logs contain "Hangup from PID: 142000 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw."
Key Process and Cause Analysis
Ceph logs are retained for 7 days, with daily compression and rotation. However, handling the SIGHUP signal directly interrupts the service.
Conclusion and Solution
The configuration in /etc/logrotate.d/ceph is used for log file rotation and has no other side effects.
The logrotate configuration indicates that Ceph logs are retained for 7 days, with daily compression and rotation.
The configuration contains the following command:
postrotate
killall -q -1 ceph-mon ceph-mds ceph-osd ceph-fuse radosgw || true
This command sends signals to those Ceph processes, which seems to shut them down.
In the killall command, -1 specifies a signal. According to signal(7) — Linux manual page, signal 1 is the SIGHUP signal. By default, handling the SIGHUP signal directly interrupts the service.