文档
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

OmniData Configuration File

Table 1 describes the configuration items in the /opt/haf-target/run/haf_user/omnidata/etc/config.properties file.

Table 1 Configuration items

Classification

Configuration Item

Default Value

Description

Maximum number of tasks

max.task.queue.size

-

Number of tasks received by OmniData. The number must be in direct proportion to the number of cores. Set the multiple based on the actual CPU performance. The default value is max (Number of available processors x 4, 4).

Size of the expression cache

compile.expression-cache-size

8192

Number of cache expressions.

Compression

compression.enabled

false

Indicates whether data is compressed.

Storage time zone

storage.timezone

-

Specify the default server time zone.

UDF plugin

external-functions-plugin.dir

/opt/haf-target/run/haf_user/omnidata/plugin

UDF directory, which exists as a folder.

Accessing Ceph/HDFS

hdfs.config.resources

/opt/haf-target/run/haf_user/omnidata/etc/hdfs-site.xml,

/opt/omnidata_dir/omnidata/etc/core-site.xml

Paths of core-site.xml and hdfs-site.xml. Use a comma (,) to separate them.

In scenario 1 of Configuring OmniData, the two files are stored in the specified directory.

Whether the HDFS is in security mode.

hdfs.authentication.type

NONE

Specify the HDFS authentication mode. The value can be NONE or KERBEROS.

Configuring the secure HDFS

hdfs.krb5.conf.path

-

Path of the krb5.cnf file. If a secure HDFS cluster is connected, configure krb5.cnf, keytab, and principal.

hdfs.krb5.keytab.path

-

Path of the keytab file.

hdfs.krb5.principal

-

User principal.

fs.hdfs.impl.disable.cache

false

Disables HDFS access to the cache.

openLooKeng registration service

discovery.service.enabled

true

Indicates whether OmniData registers with Discovery and sends status information.

internal-communication.https.required

false

Indicates whether OmniData sends HTTPS/HTTP requests to the Discovery service.

Spark registration service

omnidata.zookeeper.heartbeat.enabled

true

Indicates whether OmniData registers with ZooKeeper and sends status information.

ZooKeeper configuration

zookeeper.quorum.server

Parameters input by users

IP address of the ZooKeeper server.

zookeeper.namespace

sdi

OmniData node name registered with ZooKeeper.

zookeeper.status.node

status

Directory registered by OmniData with ZooKeeper for storing pushdown information.

zookeeper.connection.timeoutMs

15000

ZooKeeper connection timeout interval, in milliseconds.

zookeeper.session.timeoutMs

60000

ZooKeeper session timeout interval, in milliseconds.

zookeeper.retry.intervalMs

1000

ZooKeeper reconnection interval upon failure, in milliseconds.

omnidata.pushdown.threshold

0.8f

Threshold of OmniData pushdown node resources.

omnidata.status.update.interval

3

Update frequency of OmniData pushdown node resources, in seconds.

Secure ZooKeeper configuration

zookeeper.krb5.enabled

false

Indicates whether the ZooKeeper krb5 security configuration is enabled.

zookeeper.java.security.auth.login.config

-

ZooKeeper secure login configuration path.

zookeeper.krb5.conf

-

Path of the krb5.conf file of ZooKeeper. When the secure ZooKeeper is connected, configure krb5.conf, keytab, and principal.

Configuring the Spark Registration Service and Secure ZooKeeper

Currently, the Spark and openLooKeng engines use ZooKeeper and the built-in Discovery service respectively to collect and manage OmniData node information so that the engines can detect the OmniData nodes and their task quantity. When OmniData connects to the Spark engine, you need to configure the Spark registration service and secure ZooKeeper configuration in the preceding table.

The following steps show a typical configuration for OmniData to connect to the Spark engine.

  1. Open the config.properties configuration file.
    1
    vi /opt/haf-target/run/haf_user/omnidata/etc/config.properties
  2. Set the following parameters, save the settings, and exit:
    zookeeper.quorum.server=xxx.xxx.xxx.xxx:2181
hdfs.config.resources=/opt/haf-target/run/haf_user/omnidata/etc/hdfs-site.xml,/opt/haf-target/run/haf_user/omnidata/etc/core-site.xml
hdfs.authentication.type=KERBEROS
external-functions-plugin.dir=/opt/haf-target/run/haf_user/omnidata/plugin
hdfs.krb5.conf.path=/opt/haf-target/run/haf_user/omnidata/etc/krb5.conf
hdfs.krb5.keytab.path=/opt/haf-target/run/haf_user/omnidata/etc/hdfs.keytab
hdfs.krb5.principal=hdfs/server1@EXAMPLE.COM
omnidata.zookeeper.heartbeat.enabled=true
zookeeper.krb5.enabled=true
zookeeper.java.security.auth.login.config=/opt/haf-target/run/haf_user/omnidata/etc/client_jaas.conf
zookeeper.krb5.conf=/opt/haf-target/run/haf_user/omnidata/etc/krb5.conf

Configuring the openLooKeng Registration Service

When OmniData connects to the openLooKeng engine, configure the openLooKeng registration service in the preceding table. openLooKeng has an airliftconfig file which is automatically generated during the installation. discovery.uri is the service address of the Discovery service, and node.id is the ID of an OmniData node. The IDs of all OmniData nodes must be unique. The value of node.environment must be the same as that of the openLooKeng cluster.

  1. Open the config.properties configuration file.
    1
    vi /opt/haf-target/run/haf_user/omnidata/etc/config.properties
  2. Set the following parameters, save the settings, and exit:
    hdfs.config.resources=/opt/haf-target/run/haf_user/omnidata/etc/hdfs-site.xml,/opt/haf-target/run/haf_user/omnidata/etc/core-site.xml
hdfs.authentication.type=KERBEROS
external-functions-plugin.dir=/opt/haf-target/run/haf_user/omnidata/plugin
hdfs.krb5.conf.path=/opt/haf-target/run/haf_user/omnidata/etc/krb5.conf
hdfs.krb5.keytab.path=/opt/haf-target/run/haf_user/omnidata/etc/hdfs.keytab
hdfs.krb5.principal=hdfs/server1@EXAMPLE.COM
discovery.service.enabled=true

    The value of node.environment must be the same as that of the openLooKeng cluster. The value of node.id must be unique.

  3. Open the configuration file.
    1
    vi /opt/haf-target/run/haf_user/omnidata/etc/airliftconfig.properties
  4. Set the following parameters, save the settings, and exit:
    discovery.uri=http://openlookeng-coordinator-ip:openlookeng-coordinator-port
node.id=everything
node.environment=omnidata