--- title: （可选）配置ZooKeeper description: "开启ZooKeeper的安全特性可以提高系统的安全性、数据保护和可靠性，同时方便管理和监控。开启ZooKeeper的安全特性的方法有两种：开启TLS+Kerberos功能和仅开启Kerberos功能。出于安全加固考虑，建议关闭ZooKeeper服务端的AdminServer和JMX功能以增强安全性。" url: https://www.hikunpeng.com/document/detail/zh/kunpengboostkithistory/240RC5/bds/kunpengbds_omniruntime_20_0132.html sourcePath: /source/zh/kunpengboostkithistory/240RC5/bds/kunpengbds_omniruntime_20_0132.html indexId: c988129f8521d8b8d0e9e7444ceabbfd1435089973c78631867913cd46a7006a80 --- # （可选）配置ZooKeeper 开启ZooKeeper的安全特性可以提高系统的安全性、数据保护和可靠性，同时方便管理和监控。开启ZooKeeper的安全特性的方法有两种：开启TLS+Kerberos功能和仅开启Kerberos功能。出于安全加固考虑，建议关闭ZooKeeper服务端的AdminServer和JMX功能以增强安全性。 下文中提到的“$OCK_HOME”为OmniShuffle Shuffle加速组件的安装目录，用户需要根据OmniShuffle Shuffle加速组件的实际安装目录进行命令或代码的修改。 #### 开启ZooKeeper安全特性 ZooKeeper安全特性默认是开启状态，以下介绍两种开启ZooKeeper安全特性的方法： - 开启TLS+Kerberos功能。 - 仅开启Kerberos功能。 下文中提到的$OCK_HOME为OCK的安装目录，zkhostname为用户环境中ZooKeeper Server服务器的hostname，zkclihostname为用户环境中ZooKeeper Client服务器的hostname，“/usr/local/zookeeper”为ZooKeeper的安装路径。用户需要根据实际情况进行命令或代码的修改。 1. 配置“~/.bashrc”文件。 Arm平台取值为“linux-aarch64”。 ``` export OCK_BINARY_TYPE=linux-aarch64 ``` 2. 可选：生成TLS功能证书。 用户需自行提供server.crt.pem、client.crt.pem和client.pem这3个证书文件。证书存放路径可自定义，本文证书存放路径以“/home/cafile/” 为例。 a. 关闭系统历史记录功能。 生成加密口令之前建议关闭系统历史记录功能，避免密码被记录下来，可在口令生成后再打开该功能。 ``` set +o history ``` b. 生成加密口令。 对生成TLS证书和pem证书时输入的口令进行加密，获得加密口令字符串，后续配置中会用到，进入kmc_tool所在目录。（请确保加密口令的复杂度以保证证书安全。） 若开启ZooKeeper的TLS功能，则每个节点都要运行，且需根据加密口令使用用户切换用户生成。将ock运行用户获得的pem证书加密口令字符串写入ock.conf的ock.zookeeper.security.certs参数，将提交Spark任务用户获得的pem证书加密口令字符串写入ock.conf的ock.zookeeper.sdk.security.certs参数。 ``` cd $OCK_HOME /ucache/24.0.0/linux-aarch64/bin LD_LIBRARY_PATH= $OCK_HOME /ucache/24.0.0/linux-aarch64/lib/common/ ./kmc_tool 0 --encrypt //交互输入 加密口令 后按“Enter” ``` c. 打开系统历史记录功能。 ``` set -o history ``` d. 可选：将生成的TLS证书拷贝至指定目录。 对生成的TLS证书根据各用户分别拷贝至用户下指定目录，拷贝完成后修改各文件属主为用户，权限为400（若不拷贝，则需将文件赋予用户组可读权限。），各用户拷贝目录如下所示： 1. ock用户将.pem文件拷贝至用户“${OCK_HOME}/security/tls/zk_client”目录（若无此目录则先创建），拷贝完成后修改目录及文件属主为ock用户，修改目录权限为500，文件权限为400。 2. ZooKeeper用户将.jks与.pem文件拷贝至用户“${HOME}/huawei/ock/security/tls/”目录（若无此目录则先创建），拷贝完成后修改目录及文件属主为ZooKeeper用户，修改目录权限为500，文件权限为400。 3. 提交Spark任务用户将.pem文件拷贝至用户“${HOME}/huawei/ock/security/tls/”目录（若无此目录则先创建），拷贝完成后修改目录及文件属主为提交Spark任务用户，修改目录权限为500，文件权限为400。 4. 拷贝完成后删除原文件。 3. 生成Kerberos认证证书。 用户需自行提供Kerberos证书文件，参考（可选）配置Keytab和Whitelist步骤（若已经执行过请跳过该步骤）。 4. 引入ZooKeeper安全加固的JAR包。 将“$OCK_HOME/jars”目录下的JAR包拷贝到ZooKeeper根目录的“lib”目录下，若根目录下没有“lib”目录，则需要自行创建，并确保权限与ZooKeeper原生JAR包权限保持一致。 ``` ll total 432K -r-xr-x---. 1 ockadmin ockadmin 9.8K May 27 16:55 ock-broadcast-sdk-24.0.0.jar -r-xr-x---. 1 ockadmin ockadmin 24K May 27 16:55 ock-launch-cluster-24.0.0.jar -r-xr-x---. 1 ockadmin ockadmin 87K May 27 16:55 ock-shuffle-manager-24.0.0-for-spark-3.3.jar -r-xr-x---. 1 ockadmin ockadmin 23K May 27 16:55 ock-shuffle-sdk-24.0.0.jar -r-xr-x--- 1 ockadmin ockadmin 87K May 27 16:55 zk-server-auth-plugin-keytab-24.0.0-assembly.jar -r-xr-x--- 1 ockadmin ockadmin 85K May 27 16:55 zk-server-auth-plugin-tls-24.0.0-assembly.jar ``` ``` ll /usr/local/zookeeper/lib/ | grep zk -r-xr-x--- 1 zookeeperadmin ockadmin 87K May 27 16:55 zk-server-auth-plugin-keytab-24.0.0-assembly.jar -r-xr-x--- 1 zookeeperadmin ockadmin 85K May 27 16:55 zk-server-auth-plugin-tls-24.0.0-assembly.jar ``` 5. 修改ZooKeeper配置（Zookeeper Server节点）。 a. 配置Kerberos认证选项。 1. 打开zoo.cfg文件。 1 vi /usr/local/zookeeper/conf/zoo.cfg 2. 按“i”进入编辑模式，开启sasl配置。 ``` authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider jaasLoginRenew=3600000 secureClientPort=2281 ``` 3. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。 b. 配置ZooKeeper环境变量。 1. 编辑zkEnv.sh文件。 1 vi /usr/local/zookeeper/bin/zkEnv.sh 2. 按“i”进入编辑模式，添加如下内容。 - 开启TLS+Kerberos功能（其中，jks文件路径以实际为准，本文以“home/cafile/”为例，password为 2.b 使用ZooKeeper用户生成，新增zookeeper.security.kmc.config配置）。 ``` export SERVER_JVMFLAGS=" -Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory -Dzookeeper.ssl.keyStore.location=/home/cafile/server.keystore.jks -Dzookeeper.ssl.keyStore.password=*** -Dzookeeper.ssl.trustStore.location=/home/cafile/server.truststore.jks -Dzookeeper.ssl.trustStore.password=*** -Dzookeeper.ssl.trustStore.type=JKS -Dzookeeper.ssl.context.supplier.class=com.huawei.ock.zookeeper.SSLContext4Server -Dzookeeper.authProvider.ock=com.huawei.ock.zookeeper.OCKAuthenticationProvider -Dzookeeper.ssl.authProvider=ock -Dzookeeper.ssl.protocol=TLSv1.2 -Dzookeeper.ssl.ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 $SERVER_JVMFLAGS" export CLIENT_JVMFLAGS=" -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location=/home/cafile/client.keystore.jks -Dzookeeper.ssl.keyStore.password=*** -Dzookeeper.ssl.keyStore.type=JKS -Dzookeeper.ssl.trustStore.location=/home/cafile/client.truststore.jks -Dzookeeper.ssl.trustStore.password=*** -Dzookeeper.ssl.trustStore.type=JKS -Dzookeeper.ssl.context.supplier.class=com.huawei.ock.zookeeper.SSLContext4Client -Dzookeeper.authProvider.ock=com.huawei.ock.zookeeper.OCKAuthenticationProvider -Dzookeeper.ssl.authProvider=ock -Dzookeeper.ssl.protocol=TLSv1.2 -Dzookeeper.ssl.ciphersuites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 $CLIENT_JVMFLAGS" export SERVER_JVMFLAGS=" -Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_server.conf -Dzookeeper.security.kmc.config=/usr/local/zookeeper/conf/kmc.conf $SERVER_JVMFLAGS" export CLIENT_JVMFLAGS=" -Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_client.conf -Dzookeeper.security.kmc.config=/usr/local/zookeeper/conf/kmc.conf $CLIENT_JVMFLAGS" ``` - 只开启Kerberos功能。 ``` export SERVER_JVMFLAGS=" -Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_server.conf $SERVER_JVMFLAGS" export CLIENT_JVMFLAGS=" -Djava.security.auth.login.config=/usr/local/zookeeper/conf/zk_client.conf $CLIENT_JVMFLAGS" ``` 3. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。 c. 创建zk_server.conf文件，支持Kerberos认证。 1. 创建zk_server.conf文件。 ``` vi /usr/local/zookeeper/conf/zk_server.conf ``` 2. 按“i”进入编辑模式，参考内容如下（路径为 （可选）配置Keytab和Whitelist 中生成的对应zookeeper用户的keytab文件路径）。 ``` Server { com.huawei.ock.zookeeper.OCKKrb5LoginModule required useKeyTab=true keyTab="/home/Zookeeperadmin/huawei/ock/security/kdc/zookeeper_en.keytab" debug=true storeKey=true useTicketCache=false principal="zookeeper/ zkhostname @EXAMPLE.COM"; }; ``` 3. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。 d. 创建zk_client.conf文件，支持Kerberos认证。 1. 创建zk_client.conf文件。 ``` vi /usr/local/zookeeper/conf/zk_client.conf ``` 2. 按“i”进入编辑模式，参考内容如下（路径为 （可选）配置Keytab和Whitelist 中生成的提交Spark任务用户的keytab文件路径）。 ``` Client { com.huawei.ock.zookeeper.OCKKrb5LoginModule required useKeyTab=tru keyTab="/home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab" storeKey=true useTicketCache=false principal="zkcli/ zkclihostname @EXAMPLE.COM"; }; ``` 3. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。 e. 创建kmc.conf文件，支持Kerberos认证。 1. 创建kmc.conf文件。 ``` vi /usr/local/zookeeper/conf/kmc.conf ``` 2. 按“i”进入编辑模式，参考内容如下（kmc.ksf.primary.path与kmc.ksf.standby.path路径为 （可选）配置Keytab和Whitelist 中生成的对应ZooKeeper用户的ks文件路径）。 ``` kmc.ksf.primary.path=/home/Zookeeperadmin/huawei/ock/security/pmt/master/ksfa kmc.ksf.standby.path=/home/Zookeeperadmin/huawei/ock/security/pmt/standby/ksfb openssl.lib.path=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common/openssl/libssl.so crypto.lib.path=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common/openssl/libcrypto.so ``` 3. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。 f. 生效配置并重启ZooKeeper Server。 1. 修改配置后，需要运行zkEnv.sh使配置生效。 ``` zkEnv.sh ``` 2. 可选：配置LD_LIBRARY_PATH路径。 以下命令仅在启动ZooKeeper Server，以及ZooKeeper Client链接时需要配置。 ``` export LD_LIBRARY_PATH=$OCK_HOME/ucache/24.0.0/linux-aarch64/lib/common:$LD_LIBRARY_PATH ``` 3. 重启ZooKeeper Server。 ``` zkServer.sh restart ``` 4. ZooKeeper Client尝试加密连接ZooKeeper Server（注意IP地址和端口号的替换）。 ``` zkCli.sh -server IP地址 : 端口号 ``` 6. 修改OmniShuffle Shuffle加速组件配置。 ock.conf文件中配置内容参考如下，ock.conf文件路径：“$OCK_HOME/conf/ock.conf” 。关于ock.conf文件中的参数说明请参考ock.conf 。 - 开启TLS+Kerberos功能。其中，pem文件路径以实际为准，本文以“/home/ockadmin/opt/ock/security/tls/server/”为例，加密口令为使用ock用户生成。 ``` ock.zookeeper.security.enable = true ock.zookeeper.server.url = zk serverIP :2281 //若不开启TLS功能则端口号配成2181 ock.ucache.rpc.tls.ca.cert.path = /home/ockadmin/opt/ock/security/tls/server/ca.cert.pem ock.ucache.rpc.tls.key.path = /home/ockadmin/opt/ock/security/tls/server/server.private.key.pem ock.ucache.rpc.tls.cert.path = /home/ockadmin/opt/ock/security/tls/server/server.cert.pem ock.ucache.rpc.tls.key.pass.path = /home/ockadmin/opt/ock/security/tls/server/server.keypass.key ock.ucache.rpc.tls.sdk.ca.cert.path = /home/Sparkadmin/huawei/ock/security/tls/ca.cert.pem ock.ucache.rpc.tls.driver.key.path = /home/Sparkadmin/huawei/ock/security/tls/server.private.key.pem ock.ucache.rpc.tls.driver.cert.path = /home/Sparkadmin/huawei/ock/security/tls/server.cert.pem ock.ucache.rpc.tls.driver.key.pass.path = /home/Sparkadmin/huawei/ock/security/tls/server.keypass.key ock.ucache.rpc.auth.type = kerberos ock.ucache.rpc.auth.kerb.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab ock.ucache.rpc.auth.kerb.server.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab ock.ucache.rpc.auth.driver.kerb.server.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-server_en.keytab ock.ucache.rpc.auth.kerb.keytab.encrypted = true ock.ucache.rpc.auth.domain = EXAMPLE.COM // 根据实际domain修改 ock.ucache.rpc.auth.server.principle.name = ock_server ock.ucache.rpc.auth.client.principle.name = ock_client ock.ucache.rpc.auth.meta.principle.mapping = xx:server //xx为管理节点IP地址，server为hostname ock.ucache.rpc.auth.driver.principle.mapping = xx:server //xx为管理节点IP，server为hostname ock.ucache.rpc.author.type = whitelist ock.ucache.rpc.author.file.path = /home/ockadmin/opt/ock/security/authorization/whitelist_en ock.ucache.rpc.author.driver.file.path = /home/Sparkadmin/huawei/ock/security/authorization/whitelist_en ock.ucache.rpc.author.file.encrypted = true ock.ucache.kmc.ksf.primary.path = /home/ockadmin/opt/ock/security/pmt/master/ksfa ock.ucache.kmc.ksf.standby.path = /home/ockadmin/opt/ock/security/pmt/standby/ksfb ock.ucache.kmc.ksf.backup.path = /home/ockadmin/opt/ock/security/pmt/kmcbakup ock.ucache.sdk.kmc.ksf.primary.path = /home/Sparkadmin/huawei/ock/security/pmt/master/ksfa ock.ucache.sdk.kmc.ksf.standby.path = /home/Sparkadmin/huawei/ock/security/pmt/standby/ksfb ock.ucache.sdk.kmc.ksf.backup.path = /home/Sparkadmin/huawei/ock/security/pmt/kmcbakup ock.zookeeper.security.principle.name = zookeeper ock.zookeeper.security.principle.hostname = zkhostname ock.zookeeper.security.strategy = GSSAPI ock.zookeeper.security.certs = /home/ockadmin/opt/ock/security/tls/server.crt.pem,/home/ockadmin/opt/ock/security/tls/client.crt.pem,/home/ockadmin/opt/ock/security/tls/client.pem,*** ock.zookeeper.sdk.security.client.principle = zkcli/ManageNode@EXAMPLE.COM ock.zookeeper.sdk.security.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab ock.zookeeper.sdk.security.certs = /home/Sparkadmin/huawei/ock/security/tls/server.crt.pem,/home/Sparkadmin/huawei/ock/security/tls/client.crt.pem,/home/Sparkadmin/huawei/ock/security/tls/client.pem,*** ock.zookeeper.security.client.principle = zkcli/ManageNode@EXAMPLE.COM ock.zookeeper.security.client.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab ock.zookeeper.security.isKeytabEncrypt = true ``` - 仅开启Kerberos功能。 ``` ock.zookeeper.security.enable = true ock.zookeeper.server.url = zk serverIP :2181 ock.ucache.rpc.tls.ca.cert.path = /home/ockadmin/opt/ock/security/tls/server/ca.cert.pem ock.ucache.rpc.tls.key.path = /home/ockadmin/opt/ock/security/tls/server/server.private.key.pem ock.ucache.rpc.tls.cert.path = /home/ockadmin/opt/ock/security/tls/server/server.cert.pem ock.ucache.rpc.tls.key.pass.path = /home/ockadmin/opt/ock/security/tls/server/server.keypass.key ock.ucache.rpc.tls.sdk.ca.cert.path = /home/Sparkadmin/huawei/ock/security/tls/ca.cert.pem ock.ucache.rpc.tls.driver.key.path = /home/Sparkadmin/huawei/ock/security/tls/server.private.key.pem ock.ucache.rpc.tls.driver.cert.path = /home/Sparkadmin/huawei/ock/security/tls/server.cert.pem ock.ucache.rpc.tls.driver.key.pass.path = /home/Sparkadmin/huawei/ock/security/tls/server.keypass.key ock.ucache.rpc.auth.type = kerberos ock.ucache.rpc.auth.kerb.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab ock.ucache.rpc.auth.kerb.server.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab ock.ucache.rpc.auth.driver.kerb.server.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-server_en.keytab ock.ucache.rpc.auth.kerb.keytab.encrypted = true ock.ucache.rpc.auth.domain = EXAMPLE.COM // 根据实际domain修改 ock.ucache.rpc.auth.server.principle.name = ock_server ock.ucache.rpc.auth.client.principle.name = ock_client ock.ucache.rpc.auth.meta.principle.mapping = xx:server //xx为管理节点IP，server为hostname ock.ucache.rpc.auth.driver.principle.mapping = xx:server //xx为管理节点IP，server为hostname ock.ucache.rpc.author.type = whitelist ock.ucache.rpc.author.file.path = /home/ockadmin/opt/ock/security/authorization/whitelist_en ock.ucache.rpc.author.driver.file.path = /home/Sparkadmin/huawei/ock/security/authorization/whitelist_en ock.ucache.rpc.author.file.encrypted = true ock.ucache.kmc.ksf.primary.path = /home/ockadmin/opt/ock/security/pmt/master/ksfa ock.ucache.kmc.ksf.standby.path = /home/ockadmin/opt/ock/security/pmt/standby/ksfb ock.ucache.kmc.ksf.backup.path = /home/ockadmin/opt/ock/security/pmt/kmcbakup ock.ucache.sdk.kmc.ksf.primary.path = /home/Sparkadmin/huawei/ock/security/pmt/master/ksfa ock.ucache.sdk.kmc.ksf.standby.path = /home/Sparkadmin/huawei/ock/security/pmt/standby/ksfb ock.ucache.sdk.kmc.ksf.backup.path = /home/Sparkadmin/huawei/ock/security/pmt/kmcbakup ock.zookeeper.security.principle.name = zookeeper ock.zookeeper.security.principle.hostname = zkhostname ock.zookeeper.security.strategy = GSSAPI ock.zookeeper.security.certs = false ock.zookeeper.sdk.security.client.principle = zkcli/ManageNode@EXAMPLE.COM ock.zookeeper.sdk.security.client.keytab = /home/Sparkadmin/huawei/ock/security/kdc/krb5-client_en.keytab ock.zookeeper.sdk.security.certs = false ock.zookeeper.security.client.principle = zkcli/ManageNode@EXAMPLE.COM ock.zookeeper.security.client.keytab = /home/ockadmin/opt/ock/security/kdc/krb5-server_en.keytab ock.zookeeper.security.isKeytabEncrypt = true ``` 7. unset系统变量。 启动OmniShuffle Shuffle加速组件时则需要使用unset 命令使LD_LIBRARY_PATH的配置无效。 ``` unset LD_LIBRARY_PATH ``` - ock.zookeeper.server.url是配置ZooKeeper的IP地址。 - ZooKeeper单节点部署，配置示例如下。 ``` ock.zookeeper.server.url = 192.168.1.100:2181 ``` - ZooKeeper集群部署，配置示例如下。 ``` ock.zookeeper.server.url = 192.168.1.100:2181,192.168.1.101:2181,192.168.1.102:2181 ``` #### ZooKeeper安全加固 - 屏蔽JMX端口 开源的ZooKeeper允许通过JMX（java management extensions）监测ZooKeeper的节点信息、连接信息等运行情况。由于JMX的访问不需要认证和鉴权，如果内存中存在用户口令明文等敏感信息，则存在敏感信息泄露的风险，因此应关闭ZooKeeper服务端的JMX功能。 屏蔽方法：设置“JMXDISABLE”为“true”。 - 屏蔽AdminServer ZooKeeper提供管理服务AdminServer，允许通过URL进行ZooKeeper管理，但容易受到攻击。应关闭ZooKeeper服务端的AdminServer功能。 屏蔽方法：设置“-DZooKeeper.admin.enableServer”为“false”。 #### 关闭ZooKeeper安全特性 1. 编辑ock.conf文件，文件路径：“$OCK_HOME/conf/ock.conf”。 ``` vi $OCK_HOME/conf/ock.conf ``` 2. 按“i”进入编辑模式，修改如下参数配置。 ``` ock.zookeeper.server.url = IP : PORT ock.zookeeper.security.enable = false ``` 其中ock.zookeeper.server.url参数值中的IP地址需要根据用户实际情况修改，IP地址为ZooKeeper的Server端IP地址，PORT为普通端口2181。 3. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。