Setting the Spark Configuration File

After installing Spark, you need to add the Spark configuration to the OmniOperator configuration file so that services can be executed.

Add the following Spark configurations to the /opt/omni-operator/conf/omni.conf file.

Open /opt/omni-operator/conf/omni.conf.
1
vi /opt/omni-operator/conf/omni.conf

Press I to enter the insert mode and add the following Spark configurations (recommended).

# <----Spark template---->
# The default decimal rounding mode in mathematical operations is HALF_UP. It indicates that a decimal is rounded to the nearest integer. If the distances between the decimal and two adjacent integers are the same, the decimal is rounded up. DOWN indicates truncation, that is, rounding towards zero.
RoundingRule=DOWN
# Indicates whether to check for rescaling in the decimal operation result. The value can be NOT_CHECK_RESCALE (default) or CHECK_RESCALE.
CheckReScaleRule=CHECK_RESCALE
# Indicates whether to replace null characters in the replace operation. The value can be REPLACE (default) or NOT_REPLANOT_REPLACE.
# For example, in InputStr="apple", ReplaceStr="*", SearchStr="", openLooKeng replaces null characters in the middle of the letters to obtain OutputStr="*a*p*p*l*e*", whereas Spark does not, and OutputStr="apple" is obtained.
EmptySearchStrReplaceRule=NOT_REPLACE
# Indicates whether to directly convert a decimal to double data in C++. The value can be CAST (default, indicating direct conversion) or CONVERT_WITH_STRING (indicating that the decimal is converted to a character string and then to double data).
CastDecimalToDoubleRule=CONVERT_WITH_STRING
# Indicates whether to return an empty string or intercept a character string if a negative index is less than the minimum index in the Substr operation. The value can be EMPTY_STRING (default) or INTERCEPT_FROM_BEYOND.
# For example, in str="apple", strLength=5, startIndex=-7, subStringLength=3, the length of the character string apple is 5, and the third character is to be obtained from the position of index -7. The minimum negative index of apple is -4. Because -7 is less than -4, openLooKeng directly returns an empty string, whereas Spark still tries to obtain the third character from the position of index -7 and returns the first non-empty character a.
NegativeStartIndexOutOfBoundsRule=INTERCEPT_FROM_BEYOND
#Indicates whether ContainerVector is supported. The value can be SUPPORT (default ) or NOT_SUPPORT.
SupportContainerVecRule=NOT_SUPPORT
# Indicates whether the precision can be reduced when a character string is converted to a date. The value can be NOT_ALLOW_REDUCED_PRECISION (default) or ALLOW_REDUCED_PRECISION.
# For example, openLooKeng supports only the complete ISO date format. That is, the month and day cannot be omitted, for example, 1996-02-08. On the other hand, the month and day can be omitted in Spark, where 1996-02-28, 1996-02, and 1996 are all supported.
StringToDateFormatRule=ALLOW_REDUCED_PRECISION
# Indicates whether VectorBatch contains the filter column. The value can be NO_EXPR (default, indicating that the filter column is not contained) or EXPR_FILTER (indicating that the filter column is contained).
SupportExprFilterRule=EXPR_FILTER
# Indicates whether to support obtaining an element from the first element when startIndex=0 in the substr operation. The value can be IS_NOT_SUPPORT (default value. The default start index is 1, and an empty string is returned by default when startIndex=0.) or IS_SUPPORT.
ZeroStartIndexSupportRule=IS_SUPPORT
# Indicates whether to verify the expression.
ExpressionVerifyRule=NOT_VERIFY

# <----Other properties---->
# Indicates whether to enable batch processing of CodeGen functions. This option is disabled by default.
enableBatchExprEvaluate=false

Press Esc, type :wq!, and press Enter to save the file and exit.

Parent topic: Using on Spark