Setting the Spark Configuration File
- Add the following Spark configurations to the /opt/omni-operator/conf/omni.conf file.
- Open /opt/omni-operator/conf/omni.conf.
vim /opt/omni-operator/conf/omni.conf
- Press I to enter the insert mode and add the following Spark configurations (recommended).
# <----Spark template----> # The default decimal rounding mode in mathematical operations is HALF_UP. It indicates that a decimal is rounded to the nearest integer. If the distances between the decimal and two adjacent integers are the same, the decimal is rounded up. DOWN indicates truncation, that is, rounding towards zero. RoundingRule=DOWN # Whether to check for rescaling in the decimal operation result. The value can be NOT_CHECK_RESCALE (default) or CHECK_RESCALE. CheckReScaleRule=CHECK_RESCALE # Whether to replace null characters in the replace operation. The value can be REPLACE (default) or NOT_REPLANOT_REPLACE. EmptySearchStrReplaceRule=NOT_REPLACE # Whether to directly convert a decimal to double data in C++. The value can be CAST (default, indicating direct conversion) or CONVERT_WITH_STRING (indicating that the decimal is converted to a character string and then to double data). CastDecimalToDoubleRule=CONVERT_WITH_STRING # Whether to return an empty string or intercept a character string if a negative index is less than the minimum index in the Substr operation. The value can be EMPTY_STRING (default) or INTERCEPT_FROM_BEYOND. # For example, in str="apple", strLength=5, startIndex=-7, subStringLength=3, the length of the character string apple is 5, and the third character is to be obtained from the position of index -7. The minimum negative index of apple is -4. Because -7 is less than -4, openLooKeng directly returns an empty string, whereas Spark still tries to obtain the third character from the position of index -7 and returns the first non-empty character a. NegativeStartIndexOutOfBoundsRule=INTERCEPT_FROM_BEYOND #Indicates whether ContainerVector is supported. The value can be SUPPORT (default ) or NOT_SUPPORT. SupportContainerVecRule=NOT_SUPPORT # Whether the precision can be reduced when a character string is converted to a date. The value can be NOT_ALLOW_REDUCED_PRECISION (default) or ALLOW_REDUCED_PRECISION. # For example, the month and day can be omitted in Spark, where 1996-02-28, 1996-02, and 1996 are all supported. StringToDateFormatRule=ALLOW_REDUCED_PRECISION # Whether VectorBatch contains the filter column. The value can be NO_EXPR (default, indicating that the filter column is not contained) or EXPR_FILTER (indicating that the filter column is contained). SupportExprFilterRule=EXPR_FILTER # Whether to support obtaining an element from the first element when startIndex=0 in the substr operation. The value can be IS_NOT_SUPPORT (default value. The default start index is 1, and an empty string is returned by default when startIndex=0.) or IS_SUPPORT. ZeroStartIndexSupportRule=IS_SUPPORT
- Press Esc, type :wq!, and press Enter to save the file and exit.
- Open /opt/omni-operator/conf/omni.conf.
Parent topic: Using Operator Acceleration on the Spark Engine