执行Spark UDF业务
- 本小节仅使用OmniOperator算子加速UDF功能才需要进行操作。当前的OmniOperator算子加速UDF插件仅支持Simple UDF,若OmniOperator算子加速UDF不支持,会回退到原生Spark。
- SQL执行、执行计划查看及结果对比方式等,请参见执行Spark引擎业务。
执行步骤请参见执行Spark引擎业务,只需在SparkExtension的Spark-SQL命令行窗口启动命令基础上补充如下命令参数即可。
- 在spark.driver.extraClassPath参数后追加/opt/omni-operator/lib/boostkit-omniop-udf-1.2.0-aarch64.jar,追加后结果如下:
--conf spark.driver.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.2.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-1.2.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-udf-1.2.0-aarch64.jar:/opt/omni-operator/lib/dependencies/*
- 在spark.executor.extraClassPath参数后追加/opt/omni-operator/lib/boostkit-omniop-udf-1.2.0-aarch64.jar,追加后结果如下:
--conf spark.executor.extraClassPath=/opt/omni-operator/lib/boostkit-omniop-spark-3.1.1-1.2.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-bindings-1.2.0-aarch64.jar:/opt/omni-operator/lib/boostkit-omniop-udf-1.2.0-aarch64.jar:/opt/omni-operator/lib/dependencies/*
- 添加--jars /opt/omni-operator/lib/boostkit-omniop-udf-1.2.0-aarch64.jar参数。
--jars /opt/omni-operator/lib/boostkit-omniop-udf-1.2.0-aarch64.jar