OSD State Cannot Change to Up After OSD Deployment

Symptom

After a server node is restarted and OSDs are deployed, all OSDs on the node are in the In state and some OSDs cannot be started. After performing 5, the command output shows that there are available huge pages. Error information similar to the following is displayed in cephadm logs:

[2025-03-15 17:49:42.399173] --base-virtaddr=0x200000000000 [2025-03-15 17:49:42.399182] --match-allocations [2025-03-15 17:49:42.399191] --file-prefix=spdk_pid152 [2025-03-15 17:49:42.399200] ]
EAL: No free 2048 kB hugepages reported on node 1
EAL: No free 2048 kB hugepages reported on node 2
EAL: No free 2048 kB hugepages reported on node 3
EAL: No free 524288 kB hugepages reported on node 1
EAL: No free 524288 kB hugepages reported on node 2
EAL: No free 524288 kB hugepages reported on node 3
TELEMETRY: No legacy callbacks, legacy socket not created
[2025-03-15 17:50:12.491533] nvme_ctrlr.c:3238:nvme_ctrlr_process_init: *ERROR*: Initialization timed out in state 8
[2025-03-15 17:50:12.491716] nvme.c: 710:nvme_ctrlr_poll_internal: *ERROR*: Failed to initialize SSD: 0000:88:00.0
[2025-03-15 17:50:12.491743] nvme_pcie_common.c: 677:nvme_pcie_qpair_abort_trackers: *ERROR*: aborting outstanding command
[2025-03-15 17:50:12.491759] nvme_qpair.c: 248:nvme_admin_qpair_print_command: *NOTICE*: IDENTIFY (06) qid:0 cid:23 nsid:0 cdw10:00000001 cdw11:00000000 PRP1 0x2b5c991000 PRP2 0x0
[2025-03-15 17:50:12.491775] nvme_qpair.c: 452:spdk_nvme_print_completion: *NOTICE*: ABORTED - BY REQUEST (00/07) qid:0 cid:23 cdw0:0 sqhd:0000 p:0 m:0 dnr:0
[2025-03-15 17:50:12.491791] nvme_ctrlr.c:1520:nvme_ctrlr_identify_done: *ERROR*: nvme_identify_controller failed!
[2025-03-15 17:50:12.491801] nvme_ctrlr.c: 891:nvme_ctrlr_fail: *ERROR*: ctrlr 0000:88:00.0 in failed state.
2025-03-15T17:50:12.512+0800 fffbd1fb0040 -1 bdev() open failed to get nvme device with transport address 0000:88:00.0
2025-03-15T17:50:12.512+0800 fffbd1fb0040 -1 bluestore(/var/lib/ceph/osd/ceph-15/) mkfs failed, (1) Operation not permitted
2025-03-15T17:50:12.512+0800 fffbd1fb0040 -1 OSD::mkfs: ObjectStore::mkfs failed with error (1) Operation not permitted
2025-03-15T17:50:12.512+0800 fffbd1fb0040 -1  ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-15/: (1) Operation not permitted
2025-03-15 17:50:12,690 fffdf1eb4dc0 DEBUG create osd.15 with 0000:88:00.0 done
2025-03-15 17:50:12,910 fffdf1eb4dc0 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-366437b4-0181-11f0-bcfe-f82e3f2347d5.target.wants/ceph-366437b4-0181-11f0-bcfe-f82e3f2347d5@osd.9.service → /etc/systemd/system/ceph-366437b4-0181-11f0-bcfe-f82e3f2347d5@.service.
2025-03-15 17:50:13,426 fffdf1eb4dc0 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-366437b4-0181-11f0-bcfe-f82e3f2347d5.target.wants/ceph-366437b4-0181-11f0-bcfe-f82e3f2347d5@osd.8.service → /etc/systemd/system/ceph-366437b4-0181-11f0-bcfe-f82e3f2347d5@.service.

Possible Cause

The logs show that the current node cannot create objects due to no available huge pages. The actual reason is that the allocation and usage of huge pages are abnormal. Common causes include: (1) the allocated huge pages are insufficient; (2) the mount point of huge pages conflicts with the default one.

Solution

Unmount /dev/hugepages.
1
umount /dev/hugepages

Reset OSD environment configurations.

cephadm shell -v /lib/modules:/lib/modules -e DRIVER_OVERRIDE=uio_pci_generic sh /var/lib/ceph/spdk_lib/scripts/setup.sh reset

Repeat steps 2 and 3 to allocate huge pages and change the driver of NVMe devices to the user-mode driver.

echo 20480 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
cephadm shell -v /lib/modules:/lib/modules -e DRIVER_OVERRIDE=uio_pci_generic sh /var/lib/ceph/spdk_lib/scripts/setup.sh

Restart the cluster.
1 2
systemctl daemon-reload systemctl restart ceph.target
If an error message is displayed during OSD deployment and the OSDs cannot be started after the cluster is restarted, delete the OSDs and deploy them again.

Run the following commands to remove an abnormal OSD from the Ceph cluster.

[OSD_ID] indicates the ID of an OSD to be removed, for example, osd.0. [FSID] indicates the FSID of the current Ceph cluster.
1 2 3 4 5 6 7
cephadm shell ceph osd stop [OSD_ID] ceph osd out [OSD_ID] ceph osd crush remove [OSD_ID] ceph osd rm [OSD_ID] ceph orch daemon rm [OSD_ID] --force ceph auth rm [OSD_ID]
Delete the OSD configuration file on the physical machine.
1 2
exit rm -rf /var/lib/ceph/[FSID]/[OSD_ID]/

Parent topic: FAQs