Documentation
我要评分
获取效率
正确性
完整性
易理解
提交关闭

(Optional) Configuring ZooKeeper

Enabling the ZooKeeper security feature improves system and data security and facilitates management and monitoring. To enable the security feature, enable Kerberos only or Kerberos and TLS both. For security hardening, you are advised to disable the AdminServer and JMX functions on the ZooKeeper server.

The following uses $OCK_HOME only as an example. Replace it with the actual OmniShuffle installation directory.

Enabling ZooKeeper Security

The ZooKeeper security feature is enabled by default. You can enable ZooKeeper security using either of the methods:

  • Enable TLS+Kerberos.
  • Enable Kerberos only.

In the following content, $OCK_HOME indicates the directory where OCK is installed, zkhostname indicates the host name of the ZooKeeper server in the user environment, zkclihostname indicates the host name of the ZooKeeper client in the user environment, and /usr/local/zookeeper indicates the directory where ZooKeeper is installed. Replace them with the actual values as required.

  1. Configure the ~/.bashrc file.

    The value is linux-aarch64 for the Arm platform.

    export OCK_BINARY_TYPE=linux-aarch64
  2. Optional: Generate a TLS certificate.
    You need to provide the server.crt.pem, client.crt.pem, and client.pem certificate files. You can customize the certificate file path. The example path used in this document is /home/cafile/.
    1. Disable history recording.

      You are advised to disable history recording before generating an encrypted password to prevent the password from being recorded. You can enable this function after the password is generated.

      set +o history
    2. Generate an encrypted password.

      When generating the TLS and PEM certificates, encrypt the entered passwords to obtain the encrypted password character string, which will be used in subsequent configuration operations. Perform the following encryption operations in the directory for which the permission is 700: (Ensure that the encrypted password meets complexity requirements to ensure certificate security.)

      To enable the TLS function of ZooKeeper, run the following command on each node as the user of the corresponding encrypted password. Write the encrypted password of the PEM certificate obtained by the OCK run user to parameter ock.zookeeper.security.certs in the ock.conf file, and write the encrypted password of the PEM certificate obtained by the user who submits the Spark task to parameter ock.zookeeper.sdk.security.certs in the ock.conf file.

      LD_LIBRARY_PATH=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common/ 
${OCK_HOME}/ucache/${OCK_VERSION}${OCK_BINARY_TYPE}/bin/kmc_tool 0 --encrypt
// In the interactive prompt, enter the encrypted password and press Enter.
    3. Enable history recording.
      set -o history
  3. Generate a Kerberos authentication certificate.

    You need to provide the Kerberos certificate file. For details, see (Optional) Configuring the keytab and whitelist Files. (If you have performed this operation, skip this step.)

  4. Introduce the JAR packages for ZooKeeper security hardening.

    Copy the JAR packages in the $OCK_HOME/jars directory to the lib directory in the ZooKeeper root directory. If the lib directory does not exist, create one. The permission on the JAR packages must be the same as that on the native JAR packages of ZooKeeper.

    ll
total 432K
-r-xr-x---. 1 ockadmin ockadmin 9.8K May 27 16:55 ock-broadcast-sdk-24.0.0.jar
-r-xr-x---. 1 ockadmin ockadmin  24K May 27 16:55 ock-launch-cluster-24.0.0.jar
-r-xr-x---. 1 ockadmin ockadmin  87K May 27 16:55 ock-Shuffle-manager-24.0.0-for-spark-3.3.jar
-r-xr-x---. 1 ockadmin ockadmin  23K May 27 16:55 ock-Shuffle-sdk-24.0.0.jar
-r-xr-x--- 1 ockadmin ockadmin  87K May 27 16:55 zk-server-auth-plugin-keytab-24.0.0-assembly.jar
-r-xr-x--- 1 ockadmin ockadmin  85K May 27 16:55 zk-server-auth-plugin-tls-24.0.0-assembly.jar
    ll /usr/local/zookeeper/lib/ | grep  zk
-r-xr-x---  1 zookeeperadmin ockadmin    87K May 27 16:55 zk-server-auth-plugin-keytab-24.0.0-assembly.jar
-r-xr-x---  1 zookeeperadmin ockadmin    85K May 27 16:55 zk-server-auth-plugin-tls-24.0.0-assembly.jar
  5. Modify the ZooKeeper configuration (ZooKeeper server node).
    1. Configure Kerberos authentication options.
      1. Open the zoo.cfg file.
        1
        vi /usr/local/zookeeper/conf/zoo.cfg
      2. Press i to enter the insert mode and enable SASL.
        authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
secureClientPort=2281
      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    2. Configure ZooKeeper environment variables.
      1. Open the zkEnv.sh file.
        1
        vi /usr/local/zookeeper/bin/zkEnv.sh
      2. Press i to enter the insert mode and add the following content:
        • Enabling TLS+Kerberos (The following uses home/cafile/ as an example path of the JKS file. Replace it with the actual path. The password is the one generated in 2.b as the ZooKeeper user. The zookeeper.security.kmc.config configuration is added.)
          export SERVER_JVMFLAGS="
-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
-Dzookeeper.ssl.keyStore.location=/home/cafile/server.keystore.jks
-Dzookeeper.ssl.keyStore.password=***
-Dzookeeper.ssl.trustStore.location=/home/cafile/server.truststore.jks
-Dzookeeper.ssl.trustStore.password=***
-Dzookeeper.ssl.trustStore.type=JKS
-Dzookeeper.ssl.context.supplier.class=com.huawei.ock.zookeeper.SSLContext4Server
-Dzookeeper.authProvider.ock=com.huawei.ock.zookeeper.OCKAuthenticationProvider
-Dzookeeper.ssl.authProvider=ock
-Dzookeeper.ssl.protocol=TLSv1.2
-Dzookeeper.ssl.ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
$SERVER_JVMFLAGS"
export CLIENT_JVMFLAGS="
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.client.secure=true
-Dzookeeper.ssl.keyStore.location=/home/cafile/client.keystore.jks
-Dzookeeper.ssl.keyStore.password=***
-Dzookeeper.ssl.keyStore.type=JKS
-Dzookeeper.ssl.trustStore.location=/home/cafile/client.truststore.jks
-Dzookeeper.ssl.trustStore.password=***
-Dzookeeper.ssl.trustStore.type=JKS
-Dzookeeper.ssl.context.supplier.class=com.huawei.ock.zookeeper.SSLContext4Client
-Dzookeeper.authProvider.ock=com.huawei.ock.zookeeper.OCKAuthenticationProvider
-Dzookeeper.ssl.authProvider=ock
-Dzookeeper.ssl.protocol=TLSv1.2
-Dzookeeper.ssl.ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
$CLIENT_JVMFLAGS"

export SERVER_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_server.conf
-Dzookeeper.security.kmc.config=/usr/local/zookeeper/conf/kmc.conf
$SERVER_JVMFLAGS"
export CLIENT_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_client.conf
-Dzookeeper.security.kmc.config=/usr/local/zookeeper/conf/kmc.conf
$CLIENT_JVMFLAGS"
        • Enabling Kerberos only
          export SERVER_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_server.conf
$SERVER_JVMFLAGS"
export CLIENT_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_client.conf
$CLIENT_JVMFLAGS"
      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    3. Create a zk_server.conf file and set it to support Kerberos authentication.
      1. Create zk_server.conf.
        vi /usr/local/zookeeper/conf/zk_server.conf
      2. Press i to enter the insert mode. The following content is for reference: (The path is the keytab file path generated in (Optional) Configuring the keytab and whitelist Files for the ZooKeeper user.)
        Server {
  com.huawei.ock.zookeeper.OCKKrb5LoginModule required
  useKeyTab=true
  keyTab="/home/Zookeeperadmin/huawei/ock/security/kdc/zookeeper_en.keytab"
  debug=true
  storeKey=true
  useTicketCache=false
  principal="zookeeper/zkhostname@EXAMPLE.COM";
};
      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    4. Create a zk_client.conf file and set it to support Kerberos authentication.
      1. Create zk_client.conf.
        vi /usr/local/zookeeper/conf/zk_client.conf
      2. Press i to enter the insert mode. The following content is for reference (The path is the keytab file path generated in (Optional) Configuring the keytab and whitelist Files for the user who submits the Spark tasks.):
        Client {
  com.huawei.ock.zookeeper.OCKKrb5LoginModule required
  useKeyTab=tru
  keyTab="/home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab"
  storeKey=true
  useTicketCache=false
  principal="zkcli/zkclihostname@EXAMPLE.COM";
};
      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    5. Create a kmc.conf file and set it to support Kerberos authentication.
      1. Create kmc.conf.
        vi /usr/local/zookeeper/conf/kmc.conf
      2. Press i to enter the insert mode. The following content is for reference (kmc.ksf.primary.path and kmc.ksf.standby.path are the .ks file paths generated in (Optional) Configuring the keytab and whitelist Files for the ZooKeeper user.):
        kmc.ksf.primary.path=/home/Zookeeperadmin/huawei/ock/security/pmt/master/ksfa
kmc.ksf.standby.path=/home/Zookeeperadmin/huawei/ock/security/pmt/standby/ksfb

openssl.lib.path=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common/openssl/libssl.so
crypto.lib.path=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common/openssl/libcrypto.so
      3. Press Esc, type :wq!, and press Enter to save the file and exit.
    6. Make the configuration take effect and restart the ZooKeeper server.
      1. After modifying the configuration, run zkEnv.sh for the modification to take effect.
        zkEnv.sh
      2. Optional: Configure LD_LIBRARY_PATH.
        The following command is required only when the ZooKeeper server is started and when ZooKeeper client is connected.
        export LD_LIBRARY_PATH=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common:$LD_LIBRARY_PATH
      3. Restart the ZooKeeper server.
        zkServer.sh restart
      4. Connect to the ZooKeeper server from the ZooKeeper client in encrypted mode.
        zkCli.sh -server IP_address:port_number
  6. Modify the OmniShuffle configuration.
    View the ock.conf file content. The file is located in $OCK_HOME/conf/ock.conf. For descriptions about parameters in the file, see ock.conf.
    • Enable TLS+Kerberos. The following uses /home/ockadmin/opt/ock/security/tls/server/ as an example path of the .pem file. Replace it with the actual path. The password is generated by the OCK user.
      ock.zookeeper.security.enable = true
ock.zookeeper.server.url = zk serverIP:2281  // If TLS is not enabled, set the port number to 2181.

ock.ucache.rpc.tls.ca.cert.path = /home/ockadmin/opt/ock/security/tls/server/ca.cert.pem
ock.ucache.rpc.tls.key.path = /home/ockadmin/opt/ock/security/tls/server/server.private.key.pem
ock.ucache.rpc.tls.cert.path = /home/ockadmin/opt/ock/security/tls/server/server.cert.pem
ock.ucache.rpc.tls.key.pass.path = /home/ockadmin/opt/ock/security/tls/server/server.keypass.key
ock.ucache.rpc.tls.sdk.ca.cert.path = /home/Sparkadmin/huawei/ock/security/tls/ca.cert.pem
ock.ucache.rpc.tls.driver.key.path = /home/Sparkadmin/huawei/ock/security/tls/server.private.key.pem
ock.ucache.rpc.tls.driver.cert.path = /home/Sparkadmin/huawei/ock/security/tls/server.cert.pem
ock.ucache.rpc.tls.driver.key.pass.path = /home/Sparkadmin/huawei/ock/security/tls/server.keypass.key

ock.ucache.rpc.auth.type = kerberos
ock.ucache.rpc.auth.kerb.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.ucache.rpc.auth.kerb.server.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.driver.kerb.server.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.domain = EXAMPLE.COM // Replace the example value with the actual domain name.
ock.ucache.rpc.auth.server.principle.name = ock_server
ock.ucache.rpc.auth.client.principle.name = ock_client
ock.ucache.rpc.auth.meta.principle.mapping = xx:server // xx indicates the management node IP address and server indicates the hostname.
ock.ucache.rpc.auth.driver.principle.mapping = xx:server // xx indicates the management node IP address and server indicates the hostname.

ock.ucache.rpc.author.type = whitelist
ock.ucache.rpc.author.file.path = /home/ockadmin/opt/ock/security/authorization/whitelist_en
ock.ucache.rpc.author.driver.file.path = /home/Sparkadmin/huawei/ock/security/authorization/whitelist_en

ock.ucache.kmc.ksf.primary.path = /home/ockadmin/opt/ock/security/pmt/master/ksfa
ock.ucache.kmc.ksf.standby.path = /home/ockadmin/opt/ock/security/pmt/standby/ksfb
ock.ucache.kmc.ksf.backup.path = /home/ockadmin/opt/ock/security/pmt/kmcbakup

ock.ucache.sdk.kmc.ksf.primary.path = /home/Sparkadmin/huawei/ock/security/pmt/master/ksfa
ock.ucache.sdk.kmc.ksf.standby.path = /home/Sparkadmin/huawei/ock/security/pmt/standby/ksfb
ock.ucache.sdk.kmc.ksf.backup.path = /home/Sparkadmin/huawei/ock/security/pmt/kmcbakup

ock.zookeeper.security.principle.name = zookeeper
ock.zookeeper.security.principle.hostname = zkhostname
ock.zookeeper.security.strategy = GSSAPI
ock.zookeeper.security.certs = /home/ockadmin/opt/ock/security/tls/server.crt.pem,/home/ockadmin/opt/ock/security/tls/client.crt.pem,/home/ockadmin/opt/ock/security/tls/client.pem,***
ock.zookeeper.sdk.security.client.principle = zkcli/ManageNode@EXAMPLE.COM
ock.zookeeper.sdk.security.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.zookeeper.sdk.security.certs = /home/Sparkadmin/huawei/ock/security/tls/server.crt.pem,/home/Sparkadmin/huawei/ock/security/tls/client.crt.pem,/home/Sparkadmin/huawei/ock/security/tls/client.pem,***
ock.zookeeper.security.client.principle = zkcli/ManageNode@EXAMPLE.COM
ock.zookeeper.security.client.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
    • Enable Kerberos only.
      ock.zookeeper.security.enable = true
ock.zookeeper.server.url = zk serverIP:2181  

ock.ucache.rpc.tls.ca.cert.path = /home/ockadmin/opt/ock/security/tls/server/ca.cert.pem
ock.ucache.rpc.tls.key.path = /home/ockadmin/opt/ock/security/tls/server/server.private.key.pem
ock.ucache.rpc.tls.cert.path = /home/ockadmin/opt/ock/security/tls/server/server.cert.pem
ock.ucache.rpc.tls.key.pass.path = /home/ockadmin/opt/ock/security/tls/server/server.keypass.key
ock.ucache.rpc.tls.sdk.ca.cert.path = /home/Sparkadmin/huawei/ock/security/tls/ca.cert.pem
ock.ucache.rpc.tls.driver.key.path = /home/Sparkadmin/huawei/ock/security/tls/server.private.key.pem
ock.ucache.rpc.tls.driver.cert.path = /home/Sparkadmin/huawei/ock/security/tls/server.cert.pem
ock.ucache.rpc.tls.driver.key.pass.path = /home/Sparkadmin/huawei/ock/security/tls/server.keypass.key

ock.ucache.rpc.auth.type = kerberos
ock.ucache.rpc.auth.kerb.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.ucache.rpc.auth.kerb.server.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.driver.kerb.server.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.domain = EXAMPLE.COM // Replace the example value with the actual domain name.
ock.ucache.rpc.auth.server.principle.name = ock_server
ock.ucache.rpc.auth.client.principle.name = ock_client
ock.ucache.rpc.auth.meta.principle.mapping = xx:server // xx indicates the management node IP address and server indicates the hostname.
ock.ucache.rpc.auth.driver.principle.mapping = xx:server // xx indicates the management node IP address and server indicates the hostname.

ock.ucache.rpc.author.type = whitelist
ock.ucache.rpc.author.file.path = /home/ockadmin/opt/ock/security/authorization/whitelist_en
ock.ucache.rpc.author.driver.file.path = /home/Sparkadmin/huawei/ock/security/authorization/whitelist_en

ock.ucache.kmc.ksf.primary.path = /home/ockadmin/opt/ock/security/pmt/master/ksfa
ock.ucache.kmc.ksf.standby.path = /home/ockadmin/opt/ock/security/pmt/standby/ksfb
ock.ucache.kmc.ksf.backup.path = /home/ockadmin/opt/ock/security/pmt/kmcbakup

ock.ucache.sdk.kmc.ksf.primary.path = /home/Sparkadmin/huawei/ock/security/pmt/master/ksfa
ock.ucache.sdk.kmc.ksf.standby.path = /home/Sparkadmin/huawei/ock/security/pmt/standby/ksfb
ock.ucache.sdk.kmc.ksf.backup.path = /home/Sparkadmin/huawei/ock/security/pmt/kmcbakup

ock.zookeeper.security.principle.name = zookeeper
ock.zookeeper.security.principle.hostname = zkhostname
ock.zookeeper.security.strategy = GSSAPI
ock.zookeeper.security.certs = false
ock.zookeeper.sdk.security.client.principle = zkcli/ManageNode@EXAMPLE.COM
ock.zookeeper.sdk.security.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.zookeeper.sdk.security.certs = false
ock.zookeeper.security.client.principle = zkcli/ManageNode@EXAMPLE.COM
ock.zookeeper.security.client.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
  7. Unset the system variable.
    When starting OmniShuffle, you need to run the unset command to invalidate the LD_LIBRARY_PATH configuration.
    unset LD_LIBRARY_PATH
  • ock.zookeeper.server.url is IP address of the ZooKeeper node.
    • The following is an example for single-node ZooKeeper configuration:
      ock.zookeeper.server.url = 192.168.1.100:2181
    • The following is an example for ZooKeeper cluster configuration:
      ock.zookeeper.server.url = 192.168.1.100:2181,192.168.1.101:2181,192.168.1.102:2181

Hardening ZooKeeper Security

  • Disabling the JMX Port

    The open-source ZooKeeper allows Java management extensions (JMX) to monitor the running status of ZooKeeper nodes and connections. Authentication and authorization are not required for JMX access. If sensitive information such as plaintext user passwords exists in the memory, such information may be leaked. Therefore, JMX must be disabled on the ZooKeeper server.

    Method: Set JMXDISABLE to true.

  • Disabling AdminServer

    ZooKeeper provides the management service AdminServer, which allows ZooKeeper to be managed through URLs and so is vulnerable to attacks. Therefore, AdminServer must be disabled on the ZooKeeper server.

    Method: Set -DZooKeeper.admin.enableServer to false.

Disabling ZooKeeper Security

  1. Edit the ock.conf file in $OCK_HOME/conf/ock.conf.
    vi $OCK_HOME/conf/ock.conf
  2. Press i to enter the insert mode and modify the following parameters:
    ock.zookeeper.server.url = IP:PORT
ock.zookeeper.security.enable = false

    For ock.zookeeper.server.url, set IP to the actual IP address of the ZooKeeper server and PORT to 2181.

  3. Press Esc, type :wq!, and press Enter to save the file and exit.
Parent topic: Installing the Feature