Rate This Document
Findability
Accuracy
Completeness
Readability

Starting the Storage Service

This section describes how to deploy the Storage Service.

  1. Obtain the binary file and configuration file from the compilation server.
    mkdir -p /opt/3fs/{bin,etc} 
    mkdir -p /var/log/3fs 
    rsync -avz meta:/home/3fs/build/bin/storage_main /opt/3fs/bin 
    rsync -avz meta:/home/3fs/configs/{storage_main_launcher.toml,storage_main.toml,storage_main_app.toml} /opt/3fs/etc 
    rsync -avz meta:/home/3fs/deploy/systemd/storage_main.service /usr/lib/systemd/system 
    rsync -avz meta:/usr/lib64/libfdb_c.so /usr/lib64
  2. Update the storage_main_app.toml configuration file.
    1. Open /opt/3fs/etc/storage_main_app.toml.
      vim /opt/3fs/etc/storage_main_app.toml
    2. Press i to enter the insert mode and modify the configuration file as follows.
      allow_empty_node_id = true 
      node_id = 10001   	# Update the node ID. Note that the node ID of each storage node must be unique.
    3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
  3. Update the storage_main_launcher.toml configuration file.
    1. Open /opt/3fs/etc/storage_main_launcher.toml.
      vim /opt/3fs/etc/storage_main_launcher.toml
    2. Press i to enter the insert mode and modify the configuration file as follows.
      ... 
      cluster_id = "stage" 
      ... 
      [mgmtd_client] 
      mgmtd_server_addresses = ["RDMA://192.168.65.10:8000"] 
      ...
    3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
  4. Update the storage_main.toml configuration file.
    1. Open /opt/3fs/etc/storage_main.toml.
      vim /opt/3fs/etc/storage_main.toml
    2. Press i to enter the insert mode and modify the configuration file as follows.
      ... 
      [server.base.groups.listener] 
      filter_list = ['enp133s0f0np0'] # Enter the RDMA NIC name.
      listen_port = 8000 
      ... 
      [server.base.groups.listener] 
      filter_list = ['enp133s0f0np0'] # Enter the RDMA NIC name.
      listen_port = 9000 
      ... 
      [server.mgmtd] 
      mgmtd_server_address = ["RDMA://192.168.65.10:8000"] 
      ... 
      [common.monitor.reporters.monitor_collector] 
      remote_ip = "192.168.65.10:10000" 
      ... 
      [server.targets] 
      target_paths = ["/storage/data0/3fs","/storage/data1/3fs","/storage/data2/3fs","/storage/data3/3fs","/storage/data4/3fs","/storage/data5/3fs","/storage/data6/3fs","/storage/data7/3fs"] 
      ...
    3. Press Esc to exit the insert mode. Type :wq! and press Enter to save the file and exit.
  5. Update the configuration of each storage node to the Mgmtd Service.
    /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "set-config --type STORAGE --file /opt/3fs/etc/storage_main.toml"
  6. Start the service.
    systemctl start storage_main
  7. Check the service status.
    systemctl status storage_main

  8. List nodes in the cluster.
    /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "list-nodes" 

    Command output:

    # Id     Type     Status               Hostname  Pid      Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion 
    # 1      MGMTD    PRIMARY_MGMTD        meta      2281735  []    N/A                  1(UPTODATE)   250228-dev-1-999999-923bdd7c 
    # 100    META     HEARTBEAT_CONNECTED  meta      2281842  []    2025-03-12 17:01:32  1(UPTODATE)   250228-dev-1-999999-923bdd7c 
    # 10001  STORAGE  HEARTBEAT_CONNECTED  storage1  3294593  []    2025-03-12 17:38:13  1(UPTODATE)   250228-dev-1-999999-923bdd7c 
    # 10002  STORAGE  HEARTBEAT_CONNECTED  storage2  476286   []    2025-03-12 17:38:12  1(UPTODATE)   250228-dev-1-999999-923bdd7c 
    # 10003  STORAGE  HEARTBEAT_CONNECTED  storage3  2173767  []    2025-03-12 17:38:12  1(UPTODATE)   250228-dev-1-999999-923bdd7c

    If no storage nodes are displayed, wait for 1 to 3 minutes.