Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

(Optional) Configuring ZooKeeper

ZooKeeper is used to manage shuffle clusters. Enabling the ZooKeeper security feature improves system and data security and facilitates management and monitoring.

Hardening ZooKeeper Security

For security hardening, you are advised to disable the AdminServer and JMX functions on the ZooKeeper server.

  • Disabling the JMX Port

    The open-source ZooKeeper allows Java management extensions (JMX) to monitor the running status of ZooKeeper nodes and connections. Authentication and authorization are not required for JMX access. If sensitive information such as plaintext user passwords exists in the memory, such information may be leaked. Therefore, you are advised to disable JMX on the ZooKeeper server.

    Method: Set JMXDISABLE to true.

  • Disabling AdminServer

    ZooKeeper provides the management service AdminServer, which allows ZooKeeper to be managed through URLs and so is vulnerable to attacks. Therefore, AdminServer must be disabled on the ZooKeeper server.

    Method: Set -DZooKeeper.admin.enableServer to false.

(Optional) Enabling ZooKeeper Security

To enable the ZooKeeper security feature (enabled by default), you can enable the TLS+Kerberos functions or only the Kerberos function. Perform the following operations to enable the ZooKeeper security feature.

  1. Configure the ~/.bashrc file.

    The value is linux-aarch64 for the Arm platform.

    export OCK_BINARY_TYPE=linux-aarch64
  2. Optional: Generate a TLS certificate.
    You need to provide the server.crt.pem, client.crt.pem, and client.pem certificate files. You can customize the certificate file path. The example path used in this document is /home/cafile/.
    1. Disable history recording.

      You are advised to disable history recording before generating an encrypted password to prevent the password from being recorded.

      set +o history
    2. Generate an encrypted password.
      1. Encrypt the passwords entered when generating the TLS and PEM certificates to obtain the encrypted password string.

        Ensure that the encrypted password meets complexity requirements to ensure certificate security.

      2. Perform the following encryption operations in the directory whose permission is 700:
        • To enable the TLS function of ZooKeeper, run the following command on each node as the user of the corresponding encrypted password.
        • Write the encrypted password string of the PEM certificate obtained by the OCK run user to the ock.zookeeper.security.certs parameter in the ock.conf file.
        • Write the encrypted password string of the PEM certificate obtained by the Spark task submitting user to the ock.zookeeper.sdk.security.certs parameter in the ock.conf file.
        LD_LIBRARY_PATH=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common/ 
${OCK_HOME}/ucache/${OCK_VERSION}${OCK_BINARY_TYPE}/bin/kmc_tool 0 --encrypt
// In the interactive prompt, enter the encrypted password and press Enter.

        $OCK_HOME: OCK installation directory. Replace the example command or code with the actual one.

    3. Enable history recording.
      set -o history
  3. Generate a Kerberos authentication certificate.

    You need to provide the Kerberos certificate file by yourself. For details, see (Optional) Configuring the keytab and whitelist Files.

  4. Introduce the JAR packages for ZooKeeper security hardening.

    Copy the JAR packages in the $OCK_HOME/jars directory to the lib directory in the ZooKeeper root directory. If the lib directory does not exist, create it and ensure that the permission for it is the same as that for the open source ZooKeeper JAR package.

    ll
total 432K
-r-xr-x---. 1 ockadmin ockadmin 9.8K May 27 16:55 ock-broadcast-sdk-24.0.0.jar
-r-xr-x---. 1 ockadmin ockadmin  24K May 27 16:55 ock-launch-cluster-24.0.0.jar
-r-xr-x---. 1 ockadmin ockadmin  87K May 27 16:55 ock-Shuffle-manager-24.0.0-for-spark-3.3.jar
-r-xr-x---. 1 ockadmin ockadmin  23K May 27 16:55 ock-Shuffle-sdk-24.0.0.jar
-r-xr-x--- 1 ockadmin ockadmin  87K May 27 16:55 zk-server-auth-plugin-keytab-24.0.0-assembly.jar
-r-xr-x--- 1 ockadmin ockadmin  85K May 27 16:55 zk-server-auth-plugin-tls-24.0.0-assembly.jar
    ll /usr/local/zookeeper/lib/ | grep  zk
-r-xr-x---  1 zookeeperadmin ockadmin    87K May 27 16:55 zk-server-auth-plugin-keytab-24.0.0-assembly.jar
-r-xr-x---  1 zookeeperadmin ockadmin    85K May 27 16:55 zk-server-auth-plugin-tls-24.0.0-assembly.jar

    /usr/local/zookeeper: ZooKeeper installation path. Replace the example command or code with the actual one.

  5. Modify the ZooKeeper configuration (ZooKeeper server node).
    1. Configure Kerberos authentication options.
      1. Open the zoo.cfg file.
        1
        vi /usr/local/zookeeper/conf/zoo.cfg
      2. Press i to enter the insert mode and enable SASL.
        authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
secureClientPort=2281
      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    2. Configure ZooKeeper environment variables.
      1. Open the zkEnv.sh file.
        1
        vi /usr/local/zookeeper/bin/zkEnv.sh
      2. Press i to enter the insert mode and add the following content:
        • Enabling TLS+Kerberos
          The following uses home/cafile/ as an example path of the JKS file. Replace it with the actual path. The password is the one generated in 2.b as the ZooKeeper user. The zookeeper.security.kmc.config configuration is added.
          export SERVER_JVMFLAGS="
-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
-Dzookeeper.ssl.keyStore.location=/home/cafile/server.keystore.jks
-Dzookeeper.ssl.keyStore.password=***
-Dzookeeper.ssl.trustStore.location=/home/cafile/server.truststore.jks
-Dzookeeper.ssl.trustStore.password=***
-Dzookeeper.ssl.trustStore.type=JKS
-Dzookeeper.ssl.context.supplier.class=com.huawei.ock.zookeeper.SSLContext4Server
-Dzookeeper.authProvider.ock=com.huawei.ock.zookeeper.OCKAuthenticationProvider
-Dzookeeper.ssl.authProvider=ock
-Dzookeeper.ssl.protocol=TLSv1.2
-Dzookeeper.ssl.ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
$SERVER_JVMFLAGS"
export CLIENT_JVMFLAGS="
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.client.secure=true
-Dzookeeper.ssl.keyStore.location=/home/cafile/client.keystore.jks
-Dzookeeper.ssl.keyStore.password=***
-Dzookeeper.ssl.keyStore.type=JKS
-Dzookeeper.ssl.trustStore.location=/home/cafile/client.truststore.jks
-Dzookeeper.ssl.trustStore.password=***
-Dzookeeper.ssl.trustStore.type=JKS
-Dzookeeper.ssl.context.supplier.class=com.huawei.ock.zookeeper.SSLContext4Client
-Dzookeeper.authProvider.ock=com.huawei.ock.zookeeper.OCKAuthenticationProvider
-Dzookeeper.ssl.authProvider=ock
-Dzookeeper.ssl.protocol=TLSv1.2
-Dzookeeper.ssl.ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
$CLIENT_JVMFLAGS"

export SERVER_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_server.conf
-Dzookeeper.security.kmc.config=/usr/local/zookeeper/conf/kmc.conf
$SERVER_JVMFLAGS"
export CLIENT_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_client.conf
-Dzookeeper.security.kmc.config=/usr/local/zookeeper/conf/kmc.conf
$CLIENT_JVMFLAGS"
        • Enabling Kerberos only
          export SERVER_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_server.conf
$SERVER_JVMFLAGS"
export CLIENT_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_client.conf
$CLIENT_JVMFLAGS"
      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    3. Create a zk_server.conf file and set it to support Kerberos authentication.
      1. Create zk_server.conf.
        vi /usr/local/zookeeper/conf/zk_server.conf
      2. Press i to enter the insert mode and edit the file as follows:

        The path is the keytab file path generated in (Optional) Configuring the keytab and whitelist Files for the ZooKeeper user.

        Server {
  com.huawei.ock.zookeeper.OCKKrb5LoginModule required
  useKeyTab=true
  keyTab="/home/Zookeeperadmin/huawei/ock/security/kdc/zookeeper_en.keytab"
  debug=true
  storeKey=true
  useTicketCache=false
  principal="zookeeper/zkhostname@EXAMPLE.COM";
};

        zkhostname: host name of the ZooKeeper server in the user environment. Replace the example command or code with the actual one.

      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    4. Create a zk_client.conf file and set it to support Kerberos authentication.
      1. Create zk_client.conf.
        vi /usr/local/zookeeper/conf/zk_client.conf
      2. Press i to enter the insert mode and edit the file as follows:

        The path is the keytab file path generated in (Optional) Configuring the keytab and whitelist Files for the user who submits the Spark task.

        Client {
  com.huawei.ock.zookeeper.OCKKrb5LoginModule required
  useKeyTab=tru
  keyTab="/home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab"
  storeKey=true
  useTicketCache=false
  principal="zkcli/zkclihostname@EXAMPLE.COM";
};

        zkclihostname: host name of the ZooKeeper client in the user environment. Replace the example command or code with the actual one.

      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    5. Create a kmc.conf file and set it to support Kerberos authentication.
      1. Create kmc.conf.
        vi /usr/local/zookeeper/conf/kmc.conf
      2. Press i to enter the insert mode and edit the file as follows:
        kmc.ksf.primary.path and kmc.ksf.standby.path are the ks file paths generated in (Optional) Configuring the keytab and whitelist Files for the ZooKeeper user.
        kmc.ksf.primary.path=/home/Zookeeperadmin/huawei/ock/security/pmt/master/ksfa
kmc.ksf.standby.path=/home/Zookeeperadmin/huawei/ock/security/pmt/standby/ksfb

openssl.lib.path=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common/openssl/libssl.so
crypto.lib.path=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common/openssl/libcrypto.so
      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    6. Make the configuration take effect and restart the ZooKeeper server.
      1. After modifying the configuration, run zkEnv.sh for the modification to take effect.
        zkEnv.sh
      2. Configure LD_LIBRARY_PATH.

        The following command is required only when the ZooKeeper server is started and when ZooKeeper client is connected.

        export LD_LIBRARY_PATH=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common:$LD_LIBRARY_PATH
      3. Restart the ZooKeeper server.
        zkServer.sh restart
      4. Connect the ZooKeeper client to the ZooKeeper server in encrypted mode.
        zkCli.sh -server IP_address:port

        Replace the example IP address and port with the actual ones.

  6. Modify the OmniShuffle configuration.
    View the ock.conf file content. The file is located in $OCK_HOME/conf/ock.conf. For descriptions about parameters in the file, see ock.conf.
    • Enabling TLS+Kerberos
      The following uses /home/ockadmin/opt/ock/security/tls/server/ as an example path of the .pem file. Replace it with the actual path. The password is generated by the OCK user.
      ock.zookeeper.security.enable = true
ock.zookeeper.server.url = ZooKeeper_server_IP_address:2281  // If TLS is not enabled, set the port number to 2181.

ock.ucache.rpc.tls.ca.cert.path = /home/ockadmin/opt/ock/security/tls/server/ca.cert.pem
ock.ucache.rpc.tls.key.path = /home/ockadmin/opt/ock/security/tls/server/server.private.key.pem
ock.ucache.rpc.tls.cert.path = /home/ockadmin/opt/ock/security/tls/server/server.cert.pem
ock.ucache.rpc.tls.key.pass.path = /home/ockadmin/opt/ock/security/tls/server/server.keypass.key
ock.ucache.rpc.tls.sdk.ca.cert.path = /home/Sparkadmin/huawei/ock/security/tls/ca.cert.pem
ock.ucache.rpc.tls.driver.key.path = /home/Sparkadmin/huawei/ock/security/tls/server.private.key.pem
ock.ucache.rpc.tls.driver.cert.path = /home/Sparkadmin/huawei/ock/security/tls/server.cert.pem
ock.ucache.rpc.tls.driver.key.pass.path = /home/Sparkadmin/huawei/ock/security/tls/server.keypass.key

ock.ucache.rpc.auth.type = kerberos
ock.ucache.rpc.auth.kerb.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.ucache.rpc.auth.kerb.server.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.driver.kerb.server.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.domain = EXAMPLE.COM // Replace the example value with the actual domain name.
ock.ucache.rpc.auth.server.principle.name = ock_server
ock.ucache.rpc.auth.client.principle.name = ock_client
ock.ucache.rpc.auth.meta.principle.mapping = xx:server // xx indicates the management node IP address and server indicates the hostname.
ock.ucache.rpc.auth.driver.principle.mapping = xx:server // xx indicates the management node IP address and server indicates the hostname.

ock.ucache.rpc.author.type = whitelist
ock.ucache.rpc.author.file.path = /home/ockadmin/opt/ock/security/authorization/whitelist_en
ock.ucache.rpc.author.driver.file.path = /home/Sparkadmin/huawei/ock/security/authorization/whitelist_en

ock.ucache.kmc.ksf.primary.path = /home/ockadmin/opt/ock/security/pmt/master/ksfa
ock.ucache.kmc.ksf.standby.path = /home/ockadmin/opt/ock/security/pmt/standby/ksfb
ock.ucache.kmc.ksf.backup.path = /home/ockadmin/opt/ock/security/pmt/kmcbakup

ock.ucache.sdk.kmc.ksf.primary.path = /home/Sparkadmin/huawei/ock/security/pmt/master/ksfa
ock.ucache.sdk.kmc.ksf.standby.path = /home/Sparkadmin/huawei/ock/security/pmt/standby/ksfb
ock.ucache.sdk.kmc.ksf.backup.path = /home/Sparkadmin/huawei/ock/security/pmt/kmcbakup

ock.zookeeper.security.principle.name = zookeeper
ock.zookeeper.security.principle.hostname = zkhostname
ock.zookeeper.security.strategy = GSSAPI
ock.zookeeper.security.certs = /home/ockadmin/opt/ock/security/tls/server.crt.pem,/home/ockadmin/opt/ock/security/tls/client.crt.pem,/home/ockadmin/opt/ock/security/tls/client.pem,***
ock.zookeeper.sdk.security.client.principle = zkcli/ManageNode@EXAMPLE.COM
ock.zookeeper.sdk.security.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.zookeeper.sdk.security.certs = /home/Sparkadmin/huawei/ock/security/tls/server.crt.pem,/home/Sparkadmin/huawei/ock/security/tls/client.crt.pem,/home/Sparkadmin/huawei/ock/security/tls/client.pem,***
ock.zookeeper.security.client.principle = zkcli/ManageNode@EXAMPLE.COM
ock.zookeeper.security.client.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
    • Enabling Kerberos only
      ock.zookeeper.security.enable = true
ock.zookeeper.server.url = zk serverIP:2181  

ock.ucache.rpc.tls.ca.cert.path = /home/ockadmin/opt/ock/security/tls/server/ca.cert.pem
ock.ucache.rpc.tls.key.path = /home/ockadmin/opt/ock/security/tls/server/server.private.key.pem
ock.ucache.rpc.tls.cert.path = /home/ockadmin/opt/ock/security/tls/server/server.cert.pem
ock.ucache.rpc.tls.key.pass.path = /home/ockadmin/opt/ock/security/tls/server/server.keypass.key
ock.ucache.rpc.tls.sdk.ca.cert.path = /home/Sparkadmin/huawei/ock/security/tls/ca.cert.pem
ock.ucache.rpc.tls.driver.key.path = /home/Sparkadmin/huawei/ock/security/tls/server.private.key.pem
ock.ucache.rpc.tls.driver.cert.path = /home/Sparkadmin/huawei/ock/security/tls/server.cert.pem
ock.ucache.rpc.tls.driver.key.pass.path = /home/Sparkadmin/huawei/ock/security/tls/server.keypass.key

ock.ucache.rpc.auth.type = kerberos
ock.ucache.rpc.auth.kerb.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.ucache.rpc.auth.kerb.server.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.driver.kerb.server.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.domain = EXAMPLE.COM // Replace the example value with the actual domain name.
ock.ucache.rpc.auth.server.principle.name = ock_server
ock.ucache.rpc.auth.client.principle.name = ock_client
ock.ucache.rpc.auth.meta.principle.mapping = xx:server // xx indicates the management node IP address and server indicates the hostname.
ock.ucache.rpc.auth.driver.principle.mapping = xx:server // xx indicates the management node IP address and server indicates the hostname.

ock.ucache.rpc.author.type = whitelist
ock.ucache.rpc.author.file.path = /home/ockadmin/opt/ock/security/authorization/whitelist_en
ock.ucache.rpc.author.driver.file.path = /home/Sparkadmin/huawei/ock/security/authorization/whitelist_en

ock.ucache.kmc.ksf.primary.path = /home/ockadmin/opt/ock/security/pmt/master/ksfa
ock.ucache.kmc.ksf.standby.path = /home/ockadmin/opt/ock/security/pmt/standby/ksfb
ock.ucache.kmc.ksf.backup.path = /home/ockadmin/opt/ock/security/pmt/kmcbakup

ock.ucache.sdk.kmc.ksf.primary.path = /home/Sparkadmin/huawei/ock/security/pmt/master/ksfa
ock.ucache.sdk.kmc.ksf.standby.path = /home/Sparkadmin/huawei/ock/security/pmt/standby/ksfb
ock.ucache.sdk.kmc.ksf.backup.path = /home/Sparkadmin/huawei/ock/security/pmt/kmcbakup

ock.zookeeper.security.principle.name = zookeeper
ock.zookeeper.security.principle.hostname = zkhostname
ock.zookeeper.security.strategy = GSSAPI
ock.zookeeper.security.certs = false
ock.zookeeper.sdk.security.client.principle = zkcli/ManageNode@EXAMPLE.COM
ock.zookeeper.sdk.security.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.zookeeper.sdk.security.certs = false
ock.zookeeper.security.client.principle = zkcli/ManageNode@EXAMPLE.COM
ock.zookeeper.security.client.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
      ock.zookeeper.server.url is IP address of the ZooKeeper node.
      • The following is an example for single-node ZooKeeper configuration:
        ock.zookeeper.server.url = 192.168.1.100:2181
      • The following is an example for ZooKeeper cluster configuration:
        ock.zookeeper.server.url = 192.168.1.100:2181,192.168.1.101:2181,192.168.1.102:2181
  7. Unset the system variable.
    When starting OmniShuffle, you need to run the unset command to invalidate the LD_LIBRARY_PATH configuration.
    unset LD_LIBRARY_PATH

Disabling ZooKeeper Security

If the ZooKeeper security feature is not needed in your service scenario, perform the following operations to disable the ZooKeeper security feature.

  1. Open the ock.conf file. The file path is $OCK_HOME/conf/ock.conf.
    vi $OCK_HOME/conf/ock.conf
  2. Press i to enter the insert mode and modify the following parameters:
    ock.zookeeper.server.url = IP:PORT
ock.zookeeper.security.enable = false
    • ock.zookeeper.server.url: Enter the actual IP address in the parameter value.
    • IP address: ZooKeeper server IP address.
    • Port: 2181.
  3. Press Esc, type :wq!, and press Enter to save the file and exit.
Parent topic: Installing the Feature