软件升级
算法库更新版本后,可以按照以下流程进行版本升级。
- 按照获取代码的说明获取最新的软件包。
- 以大数据组件的授权用户登录服务器,将原来客户端的“/home/test/boostkit/lib/”目录下的算法库软件包,替换成最新版本。
1 2 3 4
rm -f /home/test/boostkit/lib/boostkit-* cp /opt/Spark-graph-algo-lib-1.3.0-spark2.3.2/graph-accelerator/target/boostkit-graph-acc_2.11-1.3.0-spark2.3.2.jar /home/test/boostkit/lib cp /opt/Spark-graph-algo-lib-1.3.0-spark2.3.2/graph-accelerator/target/boostkit-graph-core_2.11-1.3.0-spark2.3.2.jar /home/test/boostkit/lib cp /opt/boostkit-graph-kernel-2.11-1.3.0-spark2.3.2-aarch64.jar /home/test/boostkit/lib
- 将提交任务shell脚本中算法库软件包名替换成最新版本的包名,使用yarn-client模式启动Spark作业,替换后的shell脚本示例如下。
1 2 3 4 5 6 7 8 9 10 11 12
#!/bin/bash spark-submit \ --class com.bigdata.graph.WCEMain \ --master yarn \ --deploy-mode client \ --driver-cores 36 \ --driver-memory 50g \ --jars "lib/fastutil-8.3.1.jar,lib/boostkit-graph-acc_2.11-1.3.0-spark2.3.2.jar,lib/boostkit-graph-kernel-2.11-1.3.0-spark2.3.2-aarch64.jar,lib/boostkit-graph-core_2.11-1.3.0-spark2.3.2.jar" \ --driver-class-path "lib/graph-test.jar:lib/fastutil-8.3.1.jar:lib/snakeyaml-1.17.jar:lib/boostkit-graph-acc_2.11-1.3.0-spark2.3.2.jar:lib/boostkit-graph-kernel-2.11-1.3.0-spark2.3.2-aarch64.jar:lib/boostkit-graph-core_2.11-1.3.0-spark2.3.2.jar" \ --conf "spark.executor.extraClassPath=fastutil-8.3.1.jar:boostkit-graph-acc_2.11-1.3.0-spark2.3.2.jar:boostkit-graph-kernel-2.11-1.3.0-spark2.3.2-aarch64.jar:boostkit-graph-core_2.11-1.3.0-spark2.3.2.jar" \ ./graph-test.jar
将提交任务shell脚本中算法库软件包名替换成最新版本的包名,使用yarn-cluster模式启动Spark作业,替换后的shell脚本示例如下。1 2 3 4 5 6 7 8 9 10 11 12 13
#!/bin/bash spark-submit \ --class com.bigdata.graph.WCEMain \ --master yarn \ --deploy-mode cluster \ --driver-cores 36 \ --driver-memory 50g \ --jars "lib/fastutil-8.3.1.jar,lib/boostkit-graph-acc_2.11-1.3.0-spark2.3.2.jar,lib/boostkit-graph-kernel-2.11-1.3.0-spark2.3.2-aarch64.jar,lib/boostkit-graph-core_2.11-1.3.0-spark2.3.2.jar" \ --driver-class-path "graph-test.jar:fastutil-8.3.1.jar:snakeyaml-1.17.jar:boostkit-graph-acc_2.11-1.3.0-spark2.3.2.jar:boostkit-graph-kernel-2.11-1.3.0-spark2.3.2-aarch64.jar:boostkit-graph-core_2.11-1.3.0-spark2.3.2.jar" \ --conf "spark.yarn.cluster.driver.extraClassPath=graph-test.jar:snakeyaml-1.17.jar:boostkit-graph-kernel-2.11-1.3.0-spark2.3.2-aarch64.jar:boostkit-graph-acc_2.11-1.3.0-spark2.3.2.jar:boostkit-graph-core_2.11-1.3.0-spark2.3.2.jar" \ --conf "spark.executor.extraClassPath=fastutil-8.3.1.jar:boostkit-graph-acc_2.11-1.3.0-spark2.3.2.jar:boostkit-graph-kernel-2.11-1.3.0-spark2.3.2-aarch64.jar:boostkit-graph-core_2.11-1.3.0-spark2.3.2.jar" \ ./graph-test.jar
脚本中的语句含义如表2所示。
父主题: 软件安装部署