Adding OSDs

Object Storage Daemon (OSD) is a Ceph cluster data management service. To add an OSD on a drive, the following conditions must be met:

• The drive must have no partition.
• The drive must have no LVM status.
• The drive cannot contain a file system.
• The drive cannot contain Ceph BlueStore OSDs.
• The drive size must be greater than 5 GB.

1. Optional: Change the huge page size.
   

   To deploy SPDK, the OS must support huge pages. If the huge page size is not 2 MB, perform the following operations to change the size:

   1. Open /boot/efi/EFI/openEuler/grub.cfg.

      vim /boot/efi/EFI/openEuler/grub.cfg

   2. Press i to enter the insert mode, find the kernel startup parameter, and add default_hugepagesz=2M hugepagesz=2M hugepages=1024.
   3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
2. Configure the number of huge pages.

   1	echo 20480 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

3. Switch to the user-mode driver.

   1	cephadm shell -v /lib/modules:/lib/modules -e DRIVER_OVERRIDE=uio_pci_generic sh /var/lib/ceph/spdk_lib/scripts/setup.sh

4. Write commands in 2 and 3 to /etc/rc.d/rc.local for persistent storage and enable automatic execution upon system startup.

   1 2	vim /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local

   

5. Check available system drives.

   1	cephadm shell sh /var/lib/ceph/spdk_lib/scripts/setup.sh status

   

6. Disables automatic OSD memory adjustment.

   cephadm shell ceph config set osd osd_memory_target_autotune false

7. Add OSDs. (Each node is allowed to start its own OSDs only.)
   • Method 1: Specify only BDFs.

     1 2 3 4 5

     # Create OSDs. cephadm spdkosd --bstr 0000:82:00.0;0000:83:00.0;0000:84:00.0 init # Start the daemon process. cephadm spdkosd deploy

   • Method 2: Use the configuration file.
     

     You can use the .yaml configuration file to start a service for deploying OSDs. This mode has the following advantages:

     • You can specify devices.
     • You can specify OSD IDs.
     1. Create an osd.yaml file and add the following content to the file to specify available devices.
        1. Open the osd.yaml file.

           1	vi osd.yaml

        2. Press i to enter the insert mode and add the following content:

           1 2 3 4 5 6

           - osd_id: 0 # osd_id bdfs: "0000:82:00.0" - osd_id: 1 bdfs: "0000:83:00.0" - osd_id: 2 bdfs: "0000:84:00.0"

           

           • osd_id: OSD service ID. The value must be unique in the cluster.
           • Devices specified by the bdfs field must be taken over by the uio driver. You can check whether the driver of an NVMe device is uio_pci_generic based on the command output in 5. If not, the device is not taken over by the uio driver. In this case, run the command in 3 to switch the driver.
        3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
     2. Start the OSD service.

        1 2 3 4 5 6 7 8 9

        cephadm spdkosd --b osd.yaml create # Other startup methods: cephadm spdkosd create # Automatically selects available devices and OSD IDs, and uses the default configuration to create and start OSDs. cephadm spdkosd --b osd.yaml --c osd.conf create # Specifies required devices and configurations for creating and starting OSDs. # Startup parameter description -vv: specifies whether to save detailed OSD logs to a file when Podman starts the daemon process. The default setting is False. Example command for enabling this function: cephadm spdkosd -vv create

        

        • The /mnt/osd_xx directory is used to mount huge pages to enable SPDK. After the /mnt directory is mounted with an image, its permission may change to read-only. Verify that this directory on the physical machines has the write permission.
        • This command must be executed on ceph1 to ceph3 in sequence.
8. Change the upper limit of the OSD memory of ceph1, ceph2, and ceph3 to 20 GB.

   cephadm shell ceph config set osd/host:ceph1 osd_memory_target 20G
   cephadm shell ceph config set osd/host:ceph2 osd_memory_target 20G
   cephadm shell ceph config set osd/host:ceph3 osd_memory_target 20G

9. Check the cluster status.

   1 2	cephadm shell ceph -s

   

   1	ceph orch ps