Enabling SM Algorithms for Shuffle Secure I/Os

When using the OmniShield feature to execute Spark services, start Spark to submit tasks.

1. Run the following SQL statements in the /opt/omnishield directory and enable shuffle I/O encryption:

   1

   spark-sql --driver-cores 5 --driver-memory 20g --num-executors 6 --executor-cores 4 --executor-memory 20g --conf spark.executor.memoryOverhead=20g --master yarn --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=18g --conf spark.task.cpus=1 --conf spark.driver.maxResultSize=40g --conf spark.authenticate.secret=testSecret --conf spark.authenticate=true --conf spark.io.encryption.enabled=true --conf spark.io.encryption.commons.config.cipher.classes=com.huawei.boostkit.omnishield.cipher.SM4Cipher --conf spark.io.crypto.cipher.transformation=SM4/GCM/NOPadding --conf spark.network.crypto.enabled=true --conf spark.network.crypto.cipher=SM4/GCM/NOPadding --conf spark.driver.extraClassPath=omnishield-1.0-SNAPSHOT.jar --conf spark.executor.extraClassPath=omnishield-1.0-SNAPSHOT.jar --jars omnishield-1.0-SNAPSHOT.jar --database tpcds_bin_partitioned_varchar_orc_2 -f /automated/lib/bd/testcase/scripts/omniop/conf/Spark_OmniRuntime_ShuffleEnhance/Spark_OmniRuntime_ShuffleEnhance_002.sql

   

   • To enable shuffle drive I/O encryption, modify the following command parameters:
     • spark.io.encryption.enabled=true
     • spark.io.encryption.commons.config.cipher.classes=Customized cipher class
     • spark.io.crypto.cipher.transformation=SM4/GCM/NOPadding
   • To enable shuffle network I/O encryption, modify the following command parameters:
     • spark.authenticate.secret=Customized key
     • spark.authenticate=true
     • spark.network.crypto.enabled=true
     • spark.network.crypto.cipher=SM4/GCM/NOPadding
2. Check whether the setting of SM algorithms for shuffled I/Os has taken effect.
   1. Obtain the Executor log file.

      1	yarn logs --applicationId application_xxx > xxx.log

   2. Check whether there is SM algorithm information. If Shuffle network IO encryption will use SM4/GCM/NOPadding algorithm and Shuffle disk IO encryption will use SM4/GCM/NOPadding algorithm are displayed, the setting has taken effect.