Configuring ZooKeeper

This section describes how to enable the OmniShuffle and ZooKeeper encryption security features.

The following uses $OCK_HOME only as an example. Replace it with the actual OmniShuffle installation directory.

Disabling ZooKeeper Security

Open the ock.conf file.
File path: $OCK_HOME/conf/ock.conf
```
vim $OCK_HOME/conf/ock.conf
```
Press i to enter the insert mode and modify the following parameters:
```
ock.zookeeper.server.url = IP:PORT
ock.zookeeper.security.enable = false
```
For ock.zookeeper.server.url, set IP to the actual IP address of the ZooKeeper server and PORT to 2181.
Press Esc, type :wq!, and press Enter to save the file and exit.

Enabling ZooKeeper Security

ZooKeeper security is disabled by default and can be enabled if required.

The following describes how to enable ZooKeeper security: using TLS+Kerberos and using Kerberos.

In the following content, $OCK_HOME indicates the directory where OCK is installed, zkhostname indicates the host name of the ZooKeeper server in the user environment, zkclihostname indicates the host name of the ZooKeeper client in the user environment, and /usr/local/zookeeper indicates the directory where ZooKeeper is installed. Replace them with the actual values as required.

Configure the ~/.bashrc file.
The value is linux-aarch64 for the ARM platform.
```
export OCK_BINARY_TYPE=linux-aarch64
```
Optional: Generate a TLS certificate.
You need to provide the server.crt.pem, client.crt.pem, and client.pem certificate files. If these files are not available, do as follows to generate them. You can customize the path for storing the certificate files. In the following, /home/cafile/ is used as an example.
1. Generate a TLS certificate.
  - On the server:
    Generate a self-signed certificate for the server and add it to the server.keystore.jks file.
    In the following command, IP1 and IP2 indicate the IP addresses of the OCKD meta server and ZooKeeper server respectively. If there are multiple ZooKeeper servers, separate the IP addresses with commas (,).
    
    When running this command, you need to enter the customized certificate password. You are advised to use a complex password to ensure certificate security.
    keytool -genkey -alias example.com -keyalg RSA -keystore server.keystore.jks -keysize 2048 -ext SAN=ip:IP1,ip:IP2
    
    Import the certification file to the server.crt file.
    keytool -export -alias example.com -file server.crt -keystore server.keystore.jks
    
    Add server.crt to the client.truststore.jks file on the client.
    keytool -import -trustcacerts -alias example.com -file server.crt -keystore client.truststore.jks
  - On the client:
    Generate a self-signed certificate for the client and add it to the client.keystore.jks file.
    In the following command, IP1 and IP2 indicate the IP addresses of the OCKD meta server and ZooKeeper server respectively. If there are multiple ZooKeeper servers, separate the IP addresses with commas (,).
    
    When running this command, you need to enter the customized certificate password. You are advised to use a complex password to ensure certificate security.
    
    keytool -genkey -alias example.com -keyalg RSA -keystore client.keystore.jks -keysize 2048 -ext SAN=ip:IP1,ip:IP2
    
    Import the certification file to the client.crt file.
    keytool -export -alias example.com -file client.crt -keystore client.keystore.jks
    
    Prepare the client.pem file for ZooKeeper.
    -aes256 indicates that the client.pem file needs to be encrypted. You need to enter the encrypted password.
    keytool -importkeystore -srckeystore client.keystore.jks -destkeystore client.keystore.pfx -deststoretype PKCS12 openssl pkcs12 -in client.keystore.pfx -nocerts -aes256 -out client.pem
    
    Add client.crt to the server.truststore.jks file on the server.
    keytool -import -trustcacerts -alias example.com -file client.crt -keystore server.truststore.jks
    
    Create a CRT file in PEM format on the client.
    openssl x509 -inform DER -outform PEM -in client.crt -out client.crt.pem
    
    Create a CRT file in PEM format on the server.
    openssl x509 -inform DER -outform PEM -in server.crt -out server.crt.pem
2. Disable history recording.
  You are advised to disable history recording before generating an encrypted password to prevent the password from being recorded. You can enable this function after the password is generated. For details, see 2.a.v.
```
set +o history
```
3. Generate an encrypted password.
  Encrypt the passwords entered when the TLS and PEM certificates are generated to obtain the encrypted password character string, which will be used in subsequent configuration. Go to the directory where kmc_tool is stored. (Ensure that the encrypted password meets complexity requirements to ensure certificate security.)
  
  To enable the TLS function of ZooKeeper, run the following command on each node as the user of the corresponding encrypted password. Write the encrypted password of the PEM certificate obtained by the OCK running user to parameter ock.zookeeper.security.certs in the ock.conf file, and write the encrypted password of the PEM certificate obtained by the user who submits the Spark task to parameter ock.zookeeper.sdk.security.certs in the ock.conf file.
```
cd $OCK_HOME/ucache/23.0.0/linux-aarch64/bin
LD_LIBRARY_PATH=$OCK_HOME/ucache/23.0.0/linux-aarch64/lib/common/ ./kmc_tool 0 --encrypt stdin
(In the interactive prompt, enter the encrypted password and press Enter.)
```
4. Enable history recording.
```
set -o history
```
5. Optional: Copy the generated TLS certificate to the specified directory.
  Copy the generated TLS certificate to the specified directory of each user. After the copy is complete, change the owner of each file to the corresponding user and set the permission to 400. (If you do not copy the file, grant the read permission on the file to the user group.) The directory of each user is as follows:
  1. For the OCK user, copy the .pem file to the ${OCK_HOME}/security/tls/zk_client directory. If the directory does not exist, create it first. After the copy is complete, change the owner of the directory and file to the OCK user, the directory permission to 500, and the file permission to 400.
  2. For the ZooKeeper user, copy the .jks and .pem files to the ${HOME}/huawei/ock/security/tls/ directory. If the directory does not exist, create it first. After the copy is complete, change the owner of the directory and files to the ZooKeeper user, the directory permission to 500, and the file permission to 400.
  3. For the user who submits the Spark tasks, copy the .pem file to the ${HOME}/huawei/ock/security/tls/ directory. If the directory does not exist, create it first. After the copy is complete, change the owner of the directory and file to this user, the directory permission to 500, and the file permission to 400.
  4. Delete the original files after the copy is complete.
Generate a Kerberos authentication certificate.
Execute the kdc_distribute.sh (refer to step 7 in Configuring the Keytab File and Trustlist). If you have executed the script, skip this step.

Introduce the JAR packages for ZooKeeper security hardening.

Copy the JAR packages in the $OCK_HOME/jars directory to the lib directory in the ZooKeeper root directory. If the lib directory does not exist, create one. The permission on the JAR packages should be the same as that on the native JAR packages of ZooKeeper.

[ockadmin@master jars]$ ll
total 432K
-r-xr-x---. 1 ockadmin ockadmin  99K May 27 16:55 ock-adaptive-tuning-23.0.0-for-spark-3.1.jar
-r-xr-x---. 1 ockadmin ockadmin 9.8K May 27 16:55 ock-broadcast-sdk-23.0.0.jar
-r-xr-x---. 1 ockadmin ockadmin  24K May 27 16:55 ock-launch-cluster-23.0.0.jar
-r-xr-x---. 1 ockadmin ockadmin  87K May 27 16:55 ock-shuffle-manager-23.0.0-for-spark-3.1.jar
-r-xr-x---. 1 ockadmin ockadmin  23K May 27 16:55 ock-shuffle-sdk-23.0.0.jar
-r-xr-x---. 1 ockadmin ockadmin 6.6K May 27 16:55 ock-tuning-sdk-23.0.0.jar
-r-xr-x--- 1 ockadmin ockadmin  87K May 27 16:55 zk-server-auth-plugin-keytab-23.0.0-assembly.jar
-r-xr-x--- 1 ockadmin ockadmin  85K May 27 16:55 zk-server-auth-plugin-tls-23.0.0-assembly.jar

[Zookeeperadmin@master lib]# ll /usr/local/zookeeper/lib/ | grep  zk
-r-xr-x---  1 zookeeperadmin ockadmin    87K May 27 16:55 zk-server-auth-plugin-keytab-23.0.0-assembly.jar
-r-xr-x---  1 zookeeperadmin ockadmin    85K May 27 16:55 zk-server-auth-plugin-tls-23.0.0-assembly.jar

Modify the ZooKeeper configuration (ZooKeeper server node).

Configure Kerberos authentication options.
1. Open the zoo.cfg file.
  1
  vim /usr/local/zookeeper/conf/zoo.cfg
2. Press i to enter the insert mode and enable SASL.
```
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
secureClientPort=2281
```
3. Press Esc, type :wq!, and press Enter to save the file and exit.

Configure ZooKeeper environment variables.

Open the zkEnv.sh file.
1
vim /usr/local/zookeeper/bin/zkEnv.sh

Press i to enter the insert mode and add the following content:

Enabling TLS+Kerberos (The following uses home/cafile/ as an example path of the JKS file. Replace it with the actual path. The password is the one generated in 2.c as the ZooKeeper user. The zookeeper.security.kmc.config configuration is added.)

export SERVER_JVMFLAGS="
-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
-Dzookeeper.ssl.keyStore.location=/home/cafile/server.keystore.jks
-Dzookeeper.ssl.keyStore.password=***
-Dzookeeper.ssl.trustStore.location=/home/cafile/server.truststore.jks
-Dzookeeper.ssl.trustStore.password=***
-Dzookeeper.ssl.trustStore.type=JKS
-Dzookeeper.ssl.context.supplier.class=com.huawei.ock.zookeeper.SSLContext4Server
-Dzookeeper.authProvider.ock=com.huawei.ock.zookeeper.OCKAuthenticationProvider
-Dzookeeper.ssl.authProvider=ock
-Dzookeeper.ssl.protocol=TLSv1.2
-Dzookeeper.ssl.ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
$SERVER_JVMFLAGS"
export CLIENT_JVMFLAGS="
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.client.secure=true
-Dzookeeper.ssl.keyStore.location=/home/cafile/client.keystore.jks
-Dzookeeper.ssl.keyStore.password=***
-Dzookeeper.ssl.keyStore.type=JKS
-Dzookeeper.ssl.trustStore.location=/home/cafile/client.truststore.jks
-Dzookeeper.ssl.trustStore.password=***
-Dzookeeper.ssl.trustStore.type=JKS
-Dzookeeper.ssl.context.supplier.class=com.huawei.ock.zookeeper.SSLContext4Client
-Dzookeeper.authProvider.ock=com.huawei.ock.zookeeper.OCKAuthenticationProvider
-Dzookeeper.ssl.authProvider=ock
-Dzookeeper.ssl.protocol=TLSv1.2
-Dzookeeper.ssl.ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
$CLIENT_JVMFLAGS"

export SERVER_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_server.conf
-Dzookeeper.security.kmc.config=/usr/local/zookeeper/conf/kmc.conf
$SERVER_JVMFLAGS"
export CLIENT_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_client.conf
-Dzookeeper.security.kmc.config=/usr/local/zookeeper/conf/kmc.conf
$CLIENT_JVMFLAGS"

Enabling Kerberos only

export SERVER_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_server.conf
$SERVER_JVMFLAGS"
export CLIENT_JVMFLAGS="
-Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_client.conf
$CLIENT_JVMFLAGS"

Press Esc, type :wq!, and press Enter to save the file and exit.
Create a zk_server.conf file and set it to support Kerberos authentication.
```
vim /usr/local/zookeeper/conf/zk_server.conf
```

Press i to enter the insert mode. The following content is for reference: (The path is the keytab file path generated in Configuring the Keytab File and Trustlist for the ZooKeeper user.)

Server {
  com.huawei.ock.zookeeper.OCKKrb5LoginModule required
  useKeyTab=true
  keyTab="/home/Zookeeperadmin/huawei/ock/security/kdc/zookeeper_en.keytab"
  debug=true
  storeKey=true
  useTicketCache=false
  principal="zookeeper/zkhostname@HUAWEI.COM";
};

Press Esc, type :wq!, and press Enter to save the file and exit.
Create a zk_client.conf file and set it to support Kerberos authentication.
```
vim /usr/local/zookeeper/conf/zk_client.conf
```

Press i to enter the insert mode. The following content is for reference (The path is the keytab file path generated in Configuring the Keytab File and Trustlist for the user who submits the Spark tasks.):

Client {
  com.huawei.ock.zookeeper.OCKKrb5LoginModule required
  useKeyTab=true
  keyTab="/home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab"
  storeKey=true
  useTicketCache=false
  principal="zkcli/zkclihostname@HUAWEI.COM";
};

Press Esc, type :wq!, and press Enter to save the file and exit.
Create a kmc.conf file and set it to support Kerberos authentication.
```
vim /usr/local/zookeeper/conf/kmc.conf
```

Press i to enter the insert mode. The following content is for reference (kmc.ksf.primary.path and kmc.ksf.standby.path are the .ks file paths generated in Configuring the Keytab File and Trustlist for the ZooKeeper user.):

kmc.ksf.primary.path=/home/Zookeeperadmin/huawei/ock/security/pmt/master/ksfa
kmc.ksf.standby.path=/home/Zookeeperadmin/huawei/ock/security/pmt/standby/ksfb

openssl.lib.path=$OCK_HOME/ucache/23.0.0/linux-aarch64/lib/common/openssl/libssl.so
crypto.lib.path=$OCK_HOME/ucache/23.0.0/linux-aarch64/lib/common/openssl/libcrypto.so

Press Esc, type :wq!, and press Enter to save the file and exit.
After modifying the configuration, run zkEnv.sh for the modification to take effect.
```
zkEnv.sh
```
Optional: Configure LD_LIBRARY_PATH.
The following command is required only when the ZooKeeper server is started and when ZooKeeper client is connected.
```
export LD_LIBRARY_PATH=$OCK_HOME/ucache/23.0.0/linux-aarch64/lib/common
```
Restart the ZooKeeper server.
```
zkServer.sh restart
```
Connect to the ZooKeeper server from the ZooKeeper client in encrypted mode.
```
zkCli.sh -server IP_address:port_number
```

Modify the OmniShuffle configuration.

The configuration in the ock.conf file is as follows:

The path of the ock.conf file is $OCK_HOME/conf/ock.conf. For descriptions about parameters in the ock.conf file, see ock.conf.

Enabling TLS+Kerberos (The following uses /home/ockadmin/opt/ock/security/tls/server/ as an example path of the .pem file. Replace it with the actual path. The password is generated by the OCK user.)

ock.zookeeper.security.enable = true
ock.zookeeper.server.url = zk serverIP:2281  // If TLS is not enabled, set the port number to 2181.

ock.ucache.rpc.tls.ca.cert.path = /home/ockadmin/opt/ock/security/tls/server/ca.cert.pem
ock.ucache.rpc.tls.key.encrypted = true
ock.ucache.rpc.tls.key.path = /home/ockadmin/opt/ock/security/tls/server/server.private.key.pem
ock.ucache.rpc.tls.cert.path = /home/ockadmin/opt/ock/security/tls/server/server.cert.pem
ock.ucache.rpc.tls.key.pass.path = /home/ockadmin/opt/ock/security/tls/server/server.keypass.key
ock.ucache.rpc.tls.sdk.ca.cert.path = /home/Sparkadmin/huawei/ock/security/tls/ca.cert.pem
ock.ucache.rpc.auth.type = kerberos
ock.ucache.rpc.auth.kerb.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.ucache.rpc.auth.kerb.server.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.kerb.keytab.encrypted = true
ock.ucache.rpc.auth.domain = HUAWEI.COM
ock.ucache.rpc.auth.server.principle.name = ock_server
ock.ucache.rpc.auth.client.principle.name = ock_client

ock.ucache.rpc.author.type = whitelist
ock.ucache.rpc.author.file.path = /home/ockadmin/opt/ock/security/authorization/whitelist_en
ock.ucache.rpc.author.file.encrypted = true

ock.ucache.kmc.ksf.primary.path = /home/ockadmin/opt/ock/security/pmt/master/ksfa
ock.ucache.kmc.ksf.standby.path = /home/ockadmin/opt/ock/security/pmt/standby/ksfb
ock.ucache.kmc.ksf.backup.path = /home/ockadmin/opt/ock/security/pmt/kmcbakup

ock.ucache.sdk.kmc.ksf.primary.path = /home/Sparkadmin/huawei/ock/security/pmt/master/ksfa
ock.ucache.sdk.kmc.ksf.standby.path = /home/Sparkadmin/huawei/ock/security/pmt/standby/ksfb
ock.ucache.sdk.kmc.ksf.backup.path = /home/Sparkadmin/huawei/ock/security/pmt/kmcbakup

ock.zookeeper.security.principle.name = zookeeper
ock.zookeeper.security.principle.hostname = zkhostname
ock.zookeeper.security.strategy = GSSAPI
ock.zookeeper.security.certs = /home/ockadmin/opt/ock/security/tls/server.crt.pem,/home/ockadmin/opt/ock/security/tls/client.crt.pem,/home/ockadmin/opt/ock/security/tls/client.pem,***
ock.zookeeper.sdk.security.client.principle = zkcli/master@HUAWEI.COM
ock.zookeeper.sdk.security.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.zookeeper.sdk.security.certs = /home/Sparkadmin/huawei/ock/security/tls/server.crt.pem,/home/Sparkadmin/huawei/ock/security/tls/client.crt.pem,/home/Sparkadmin/huawei/ock/security/tls/client.pem,***
ock.zookeeper.security.client.principle = zkcli/master@HUAWEI.COM
ock.zookeeper.security.client.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
ock.zookeeper.security.isKeytabEncrypt = true

Enabling Kerberos only

ock.zookeeper.security.enable = true
ock.zookeeper.server.url = zk serverIP:2181  // If TLS is not enabled, set the port number to 2181.

ock.ucache.rpc.tls.ca.cert.path = /home/ockadmin/opt/ock/security/tls/server/ca.cert.pem
ock.ucache.rpc.tls.key.encrypted = true
ock.ucache.rpc.tls.key.path = /home/ockadmin/opt/ock/security/tls/server/server.private.key.pem
ock.ucache.rpc.tls.cert.path = /home/ockadmin/opt/ock/security/tls/server/server.cert.pem
ock.ucache.rpc.tls.key.pass.path = /home/ockadmin/opt/ock/security/tls/server/server.keypass.key
ock.ucache.rpc.tls.sdk.ca.cert.path = /home/Sparkadmin/huawei/ock/security/tls/ca.cert.pem
ock.ucache.rpc.auth.type = kerberos
ock.ucache.rpc.auth.kerb.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.ucache.rpc.auth.kerb.server.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
ock.ucache.rpc.auth.kerb.keytab.encrypted = true
ock.ucache.rpc.auth.domain = HUAWEI.COM
ock.ucache.rpc.auth.server.principle.name = ock_server
ock.ucache.rpc.auth.client.principle.name = ock_client

ock.ucache.rpc.author.type = whitelist
ock.ucache.rpc.author.file.path = /home/ockadmin/opt/ock/security/authorization/whitelist_en
ock.ucache.rpc.author.file.encrypted = true

ock.ucache.kmc.ksf.primary.path = /home/ockadmin/opt/ock/security/pmt/master/ksfa
ock.ucache.kmc.ksf.standby.path = /home/ockadmin/opt/ock/security/pmt/standby/ksfb
ock.ucache.kmc.ksf.backup.path = /home/ockadmin/opt/ock/security/pmt/kmcbakup

ock.ucache.sdk.kmc.ksf.primary.path = /home/Sparkadmin/huawei/ock/security/pmt/master/ksfa
ock.ucache.sdk.kmc.ksf.standby.path = /home/Sparkadmin/huawei/ock/security/pmt/standby/ksfb
ock.ucache.sdk.kmc.ksf.backup.path = /home/Sparkadmin/huawei/ock/security/pmt/kmcbakup

ock.zookeeper.security.principle.name = zookeeper
ock.zookeeper.security.principle.hostname = zkhostname
ock.zookeeper.security.strategy = GSSAPI
ock.zookeeper.security.certs = false
ock.zookeeper.sdk.security.client.principle = zkcli/master@HUAWEI.COM
ock.zookeeper.sdk.security.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab
ock.zookeeper.sdk.security.certs = false
ock.zookeeper.security.client.principle = zkcli/master@HUAWEI.COM
ock.zookeeper.security.client.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab
ock.zookeeper.security.isKeytabEncrypt = true

Unset the system variable.
When starting OCKD, you need to run the unset command to invalidate the LD_LIBRARY_PATH configuration.
```
unset LD_LIBRARY_PATH
```

Hardening ZooKeeper Security

ZooKeeper provides the management service AdminServer, which allows the ZooKeeper to be managed through URLs and so is vulnerable to attacks. You are advised to disable AdminServer on the ZooKeeper server.

The open-source ZooKeeper allows Java management extensions (JMX) to monitor the running status of ZooKeeper nodes and connections. Authentication and authorization are not required for JMX access. If sensitive information such as plaintext user passwords exists in the memory, such information may be leaked. Therefore, you are advised to disable JMX on the ZooKeeper server.

Disabling the JMX Port

Method: Set JMXDISABLE to true.

Disabling AdminServer

ZooKeeper provides the management service AdminServer, which allows the ZooKeeper to be managed through URLs and so is vulnerable to attacks. Therefore, AdminServer must be disabled on the ZooKeeper server.

Method: Set -DZooKeeper.admin.enableServer to false.

Enable the security feature by following instructions in Enabling ZooKeeper Security.

Parent topic: Installation Requirements