Installing the UDF Plugin

OmniOperator UDFs are used to execute UDFs written based on the Hive UDF framework. The OmniOperator UDF plugin supports only simple UDFs. To use OmniOperator UDFs, install the UDF plugin and configure the expression processing method.

Prerequisites

OmniOperator UDFs support expression row-by-row processing and batch processing. The two methods can be switched using the configuration file.

Configuring the OmniOperator UDF plugin requires related JAR packages and configuration files, including udf.zip, conf.zip, and udf.properties. The udf.zip package contains all UDF class files, the conf.zip file includes the configuration file on which OmniOperator UDFs depend, and udf.properties is the OmniOperator UDF configuration file. The following uses the udfName1 and udfName2 functions as an example to describe the content format of the udf.properties file.

udfName1 com.huawei.udf.UdfName1
udfName2 com.huawei.udf.UdfName2

Upload the previously mentioned packages to the /opt/omni-operator/hive-udf directory on management and compute nodes.

Use the batch distribution function to select all nodes and click Config Service. Select File Transfer, click Add a Task, set the task parameters according to Table 1 to Table 3, and click Save.

**Table 1** Task 1 parameters
Parameter	Description
Task Name	Copying the udf.zip software package
Task Description	Copy the udf.zip software package to all nodes.
Transfer Direction	From Local to Remote
Local Path	Directory where the local installation package is stored
Remote Path	/opt/omni-operator/hive-udf

**Table 2** Task 2 parameters
Parameter	Description
Task Name	Copying the conf.zip software package
Task Description	Copy the conf.zip software package to all nodes.
Transfer Direction	From Local to Remote
Local Path	Directory where the local installation package is stored
Remote Path	/opt/omni-operator/hive-udf

**Table 3** Task 3 parameters
Parameter	Description
Task Name	Copying the udf.properties file
Task Description	Copy the udf.properties file to all nodes.
Transfer Direction	From Local to Remote
Local Path	Directory where the local installation package is stored
Remote Path	/opt/omni-operator/hive-udf

Decompress udf.zip and conf.zip and set environment variables.

Select Command Execution, click Add a Task, set the task parameters according to Table 4 to Table 5, and click Save.

**Table 4** Task 4 parameters
Parameter	Description
Task Name	Decompressing the udf.zip and conf.zip software packages
Task Description	Decompressing the udf.zip and conf.zip software packages
Run Directory	/opt/omni-operator/hive-udf
Run Command	unzip -o udf.zip;rm -f udf.zip;unzip -o conf.zip;rm -f conf.zip;

The package name can be customized. The udf.zip package is only an example.

**Table 5** Task 5 parameters
Parameter	Description
Task Name	Setting OmniOperator UDF environment variables
Task Description	Setting OmniOperator UDF environment variables on all nodes
Run Directory	/opt/omni-operator
Run Command	sed -i '$a export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/aarch64/server/:$LD_LIBRARY_PATH' ~/.bashrc;source ~/.bashrc;

Export the service flow and save it to your local PC. You can import service flows to quickly install OmniOperator.
1. Click Export in the upper right corner. A dialog box is displayed for you to select a path for saving the template.
2. Name the template OmniOperator UDF Plugin Installation and Configuration Workflow and click Save.
Execute the service flow. The procedure is complete when the execution is successful.

Installing the UDF Plugin Row-by-Row Processing

After installing the OmniOperator UDF plugin, you can add configuration items to enable UDFs to enter the row-by-row processing process. Therefore, you need to install the OmniOperator UDF plugin first. For details, see Installing the UDF Plugin.

Configure row-by-row processing parameters in the omni.conf file on all nodes.

Use the batch distribution function and select Command Execution. Click Add a Task, set the task parameters according to Table 6 and Table 7, and click Save.

**Table 6** Task 1 parameters
Parameter	Description
Task Name	Creating an /opt/omni-operator/conf directory
Task Description	Create an /opt/omni-operator/conf directory. If this directory already exists, delete it.
Run Directory	/opt/omni-operator
Run Command	mkdir -p /opt/omni-operator/conf; sed -i '/enableBatchExprEvaluate/d' /opt/omni-operator/conf/omni.conf; sed -i '/hiveUdfPropertyFilePath/d' /opt/omni-operator/conf/omni.conf; sed -i '/hiveUdfDir/d' /opt/omni-operator/conf/omni.conf;

**Table 7** Task 2 parameters
Parameter	Description
Task Name	Configuring row-by-row processing parameters
Task Description	Configure row-by-row processing parameters in the omni.conf file on all nodes.
Run Directory	/opt/omni-operator/conf
Run Command	echo "enableBatchExprEvaluate=false" >> /opt/omni-operator/conf/omni.conf; echo "hiveUdfPropertyFilePath=/opt/omni-operator/hive-udf/udf.properties" >> /opt/omni-operator/conf/omni.conf; echo "hiveUdfDir=/opt/omni-operator/hive-udf/udf" >> /opt/omni-operator/conf/omni.conf;

Export the service flow and save it to your local PC. You can import service flows to quickly install other components.
1. Click Export in the upper right corner. The Select the path where the template is saved window is displayed.
2. Name the template OmniOperator UDF Plugin Row-by-Row Processing Installation and Configuration Workflow and click Save.
Execute the service flow. The procedure is complete when the execution is successful.

Installing the UDF Plugin Batch Processing

After installing row-by-row processing, perform the following steps on the management node and all compute nodes:

Configure batch processing parameters in the omni.conf file on all nodes.

Use the batch distribution function and select Command Execution. Click Add a Task, set the task parameters according to Table 8, and click Save.

**Table 8** Task 1 parameters
Parameter	Description
Task Name	Configuring batch processing parameters
Task Description	Configure batch processing parameters in the omni.conf file on all nodes.
Run Directory	/opt/omni-operator/conf
Run Command	sed -i '/enableBatchExprEvaluate/d' /opt/omni-operator/conf/omni.conf; echo "enableBatchExprEvaluate=true" >> /opt/omni-operator/conf/omni.conf;

If enableBatchExprEvaluate is set to true, OmniOperator batch processing is enabled. If it is set to false, OmniOperator batch processing is disabled. The default value is false.

Export the service flow and save it to your local PC. You can import service flows to quickly install other components.
1. Click Export in the upper right corner.
  The Select the path where the template is saved window is displayed.
2. Name the template OmniOperator UDF Plugin Batch Processing Installation and Configuration Workflow and click Save.
Execute the service flow. The procedure is complete when the execution is successful.

Parent topic: Installation Using SmartKit