Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Installing the UDF Plugin

Prerequisites

  • Operations in this section are required only when operator acceleration UDFs are used. Currently, the operator acceleration UDF plugin supports only simple UDFs. It is used to execute UDFs compiled based on the Hive UDF framework.
  • Operator acceleration UDFs support expression row-by-row processing and batch processing. The two methods can be switched using the configuration file.

Configuring UDFs for operator pushdown requires related JAR packages and configuration files, including udf.zip, conf.zip, and udf.properties. The udf.zip package contains all UDF class files, the conf.zip package includes the configuration files on which the UDFs depend, and udf.properties is the UDF configuration file. The following uses the udfName1 and udfName2 functions as an example to describe the content format of the udf.properties file:

  1. Upload the previously mentioned packages to the /opt/omni-operator/hive-udf directory on management and compute nodes.
    Use the batch distribution function to select all nodes and click Config Service. Select File Transfer, click Add a Task, set the task parameters according to Table 1 to Table 3, and click Save.
    Table 1 Task 1 parameters

    Parameter

    Description

    Task Name

    Copying the udf.zip software package

    Task Description

    Copy the udf.zip software package to all nodes.

    Transfer Direction

    From Local to Remote

    Local Path

    Directory where the local installation package is stored

    Remote Path

    /opt/omni-operator/hive-udf
    Table 2 Task 2 parameters

    Parameter

    Description

    Task Name

    Copying the conf.zip software package

    Task Description

    Copy the conf.zip software package to all nodes.

    Transfer Direction

    From Local to Remote

    Local Path

    Directory where the local installation package is stored

    Remote Path

    /opt/omni-operator/hive-udf
    Table 3 Task 3 parameters

    Parameter

    Description

    Task Name

    Copying the udf.properties file

    Task Description

    Copy the udf.properties file to all nodes.

    Transfer Direction

    From Local to Remote

    Local Path

    Directory where the local installation package is stored

    Remote Path

    /opt/omni-operator/hive-udf
  2. Decompress udf.zip and conf.zip and set environment variables.
    Select Command Execution, click Add a Task, set the task parameters according to Table 4 to Table 5, and click Save.
    Table 4 Task 4 parameters

    Parameter

    Description

    Task Name

    Decompressing the udf.zip and conf.zip software packages

    Task Description

    Decompressing the udf.zip and conf.zip software packages

    Run Directory

    /opt/omni-operator/hive-udf

    Run Command

    unzip -o udf.zip;rm -f udf.zip;unzip -o conf.zip;rm -f conf.zip;

    You can customize the example name of the udf.zip package based on your service requirements.

    Table 5 Task 5 parameters

    Parameter

    Description

    Task Name

    Configuring UDF environment variables for operator acceleration

    Task Description

    Set UDF environment variables for operator acceleration on all nodes.

    Run Directory

    /opt/omni-operator

    Run Command

    sed -i '$a export LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/aarch64/server/:$LD_LIBRARY_PATH' ~/.bashrc;source ~/.bashrc;
  3. Export the service flow and save it to your local PC. You can import service flows to quickly install OmniOperator.
    1. Click Export in the upper right corner. A dialog box is displayed for you to select a path for saving the template.
    2. Name the template Operator Acceleration UDF Plugin Installation and Configuration Workflow and click Save.
  4. Execute the service flow. The procedure is complete when the execution is successful.

Installing the UDF Plugin Row-by-Row Processing

After the operator acceleration UDF plugin is installed, you can add configuration items to enable UDFs to enter the row-by-row processing process. Therefore, you need to install the operator acceleration UDF plugin first. For details, see Installing the UDF Plugin.

  1. Configure row-by-row processing parameters in the omni.conf file on all nodes.
    Use the batch distribution function and select Command Execution. Click Add a Task, set the task parameters according to Table 6 and Table 7, and click Save.
    Table 6 Task 1 parameters

    Parameter

    Description

    Task Name

    Creating an /opt/omni-operator/conf directory

    Task Description

    Create an /opt/omni-operator/conf directory. If this directory already exists, delete it.

    Run Directory

    /opt/omni-operator

    Run Command

    mkdir -p /opt/omni-operator/conf;

    sed -i '/enableBatchExprEvaluate/d' /opt/omni-operator/conf/omni.conf;

    sed -i '/hiveUdfPropertyFilePath/d' /opt/omni-operator/conf/omni.conf;

    sed -i '/hiveUdfDir/d' /opt/omni-operator/conf/omni.conf;
    Table 7 Task 2 parameters

    Parameter

    Description

    Task Name

    Configuring row-by-row processing parameters

    Task Description

    Configure row-by-row processing parameters in the omni.conf file on all nodes.

    Run Directory

    /opt/omni-operator/conf

    Run Command

    echo "enableBatchExprEvaluate=false" >> /opt/omni-operator/conf/omni.conf;

    echo "hiveUdfPropertyFilePath=/opt/omni-operator/hive-udf/udf.properties" >> /opt/omni-operator/conf/omni.conf;

    echo "hiveUdfDir=/opt/omni-operator/hive-udf/udf" >> /opt/omni-operator/conf/omni.conf;
  2. Export the service flow and save it to your local PC. You can import service flows to quickly install other components.
    1. Click Export in the upper right corner. The Select the path where the template is saved window is displayed.
    2. Name the template Operator Acceleration UDF Plugin Row-by-Row Processing Installation and Configuration Workflow and click Save.
  3. Execute the service flow. The procedure is complete when the execution is successful.

Installing the UDF Plugin Batch Processing

After installing row-by-row processing, perform the following steps on the management node and all compute nodes:

  1. Configure batch processing parameters in the omni.conf file on all nodes.
    Use the batch distribution function and select Command Execution. Click Add a Task, set the task parameters according to Table 8, and click Save.
    Table 8 Task 1 parameters

    Parameter

    Description

    Task Name

    Configuring batch processing parameters

    Task Description

    Configure batch processing parameters in the omni.conf file on all nodes.

    Run Directory

    /opt/omni-operator/conf

    Run Command

    sed -i '/enableBatchExprEvaluate/d' /opt/omni-operator/conf/omni.conf;

    echo "enableBatchExprEvaluate=true" >> /opt/omni-operator/conf/omni.conf;

    If enableBatchExprEvaluate is set to true, operator acceleration batch processing is enabled. If it is set to false, operator acceleration batch processing is disabled. The default value is false.

  2. Export the service flow and save it to your local PC. You can import service flows to quickly install other components.
    1. Click Export in the upper right corner.

      The Select the path where the template is saved window is displayed.

    2. Name the template Operator Acceleration UDF Plugin Batch Processing Installation and Configuration Workflow and click Save.
  3. Execute the service flow. The procedure is complete when the execution is successful.