--- title: （可选）安装并执行Spark UDF业务 description: "仅在使用UDF功能的情况下，才需要执行Spark UDF业务。在运行UDF之前，需要确保Spark引擎已经启动并正常运行。" url: https://www.hikunpeng.com/document/detail/zh/kunpengboostkithistory/2530/bds/kunpengbds_omniruntime_20_0222.html sourcePath: /source/zh/kunpengboostkithistory/2530/bds/kunpengbds_omniruntime_20_0222.html indexId: 567e20c869cc92a77f934f677ee0e5b7de15df6040dd4a1bd302195c7f10857578 --- # （可选）安装并执行Spark UDF业务 仅在使用UDF功能的情况下，才需要执行Spark UDF业务。在运行UDF之前，需要确保Spark引擎已经启动并正常运行。 当前的UDF插件仅支持Simple UDF。若OmniOperator算子加速特性无法加速UDF，会回退到Spark本身的UDF实现方式。 #### 安装Spark UDF插件 如需使用特定的数据处理操作，并且需要使用UDF功能，才需要安装Spark UDF插件。 1. 请确认安装Spark UDF插件前已经按照安装SparkExtension完成Spark引擎的安装。 2. 将UDF依赖的JAR放置到HDFS的“/user/hive-udf”目录下。 “/user/hive-udf”目录用户可自行定义。 UDF依赖的JAR需要用户自行提供。 3. 在集群的管理节点注册Hive UDF。 注册方式请参见Spark集成Hive UDF官方文档(https://spark.apache.org/docs/latest/sql-ref-functions-udf-hive.html)。 #### 执行Spark UDF业务 SQL执行、执行计划查看及结果对比方式等，请参见执行Spark引擎业务。 执行步骤请参见执行Spark引擎业务，只需在SparkExtension的Spark-SQL命令行启动命令基础上，补充如下命令参数。 1. 在spark.driver.extraClassPath参数后追加/opt/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar。 - Spark 3.1.1版本追加后结果如下。 1 --conf spark.driver.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-2.0.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-2.0.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar:/opt/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar:/opt/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.1.1-2.0.0.jar - Spark 3.3.1版本追加后结果如下。 1 --conf spark.driver.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.3.1-2.0.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-2.0.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar:/opt/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar:/opt/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.3.1-2.0.0.jar - Spark 3.4.3版本追加后结果如下。 1 --conf spark.driver.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.4.3-2.0.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-2.0.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar:/opt/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar:/opt/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.4.3-2.0.0.jar - Spark 3.5.2版本追加后结果如下。 1 --conf spark.driver.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.5.2-2.0.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-2.0.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar:/opt/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar:/opt/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.5.2-2.0.0.jar 2. 在spark.executor.extraClassPath参数后追加/opt/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar。 - Spark 3.1.1版本追加后结果如下。 1 --conf spark.executor.extraClassPath='${PWD}/omni/omni-operator/lib/boostkit-omniop-spark-3.1.1-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/boostkit-omniop-bindings-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar':'${PWD}/omni/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.1.1-2.0.0.jar' - Spark 3.3.1版本追加后结果如下。 1 --conf spark.executor.extraClassPath='${PWD}/omni/omni-operator/lib/boostkit-omniop-spark-3.3.1-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/boostkit-omniop-bindings-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar':'${PWD}/omni/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.3.1-2.0.0.jar' - Spark 3.4.3版本追加后结果如下。 1 --conf spark.executor.extraClassPath='${PWD}/omni/omni-operator/lib/boostkit-omniop-spark-3.4.3-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/boostkit-omniop-bindings-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar':'${PWD}/omni/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.4.3-2.0.0.jar' - Spark 3.5.2版本追加后结果如下。 1 --conf spark.executor.extraClassPath='${PWD}/omni/omni-operator/lib/boostkit-omniop-spark-3.5.2-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/boostkit-omniop-bindings-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/boostkit-omniop-udf-2.0.0-aarch64.jar':'${PWD}/omni/omni-operator/lib/dependencies/protobuf-java-3.15.8.jar':'${PWD}/omni/omni-operator/lib/dependencies/boostkit-omniop-native-reader-3.5.2-2.0.0.jar'