--- title: 算子和表达式支持情况 description: "介绍Spark 3.3.1引擎在使用OmniOperator算子加速特性时，OmniOperator算子加速特性支持的支持的算子、表达式及其数据类型列表。" url: https://www.hikunpeng.com/document/detail/zh/kunpengboostkithistory/2400/bds/kunpengbds_omniruntime_20_0217.html sourcePath: /source/zh/kunpengboostkithistory/2400/bds/kunpengbds_omniruntime_20_0217.html indexId: db4e1ae05f75ddb44bfa21fe642be5cd4ba089f4a37d810d085ea02394aabfda78 --- # 算子和表达式支持情况 介绍Spark 3.3.1引擎在使用OmniOperator算子加速特性时，OmniOperator算子加速特性支持的支持的算子、表达式及其数据类型列表。 - 本节描述OmniOperator算子加速特性支持的算子和表达式仅针对Spark 3.3.1引擎。 - 支持算子列表 和 支持表达式列表 中仅描述了OmniOperator算子加速支持或涉及的数据类型，未展示的数据类型（BYTE/FLOAT/BINARY/ARRAY/MAP/STRUCT/CALENDAR/UDT）是OmniOperator算子加速不支持的。 - 如果使用OmniOperator算子加速不支持的算子和表达式，会导致执行计划回退为原生执行，对性能会有影响。 算子和表达式支持列表中使用符号表示它们的支持状态，符号说明如表1所示。 **表1 符号支持状态** | 状态 | 说明 | | --- | --- | | S | 表示OmniOperator算子加速支持。 | | PS | 表示OmniOperator算子加速部分支持，存在一些限定条件。 | | NS | 表示OmniOperator算子加速不支持。 | | NA | 表示不涉及，Spark原生也没有此输入场景。 | | NA\-2 | 表示Spark上下文函数，Spark内置实现，不涉及OmniOperator算子加速。 | | [Blank Cell] | 表示不适用或需要确认的情况。 | #### 支持算子列表 Spark引擎使用OmniOperator算子加速特性支持的算子情况如表2所示。 **表2 支持算子列表** | 原生算子名称 | OmniOperator算子加速算子名称 | BOOLEAN | INT | LONG | DOUBLE | STRING | CHAR | VARCHAR | DATE | DECIMAL | SHORT | TIMESTAMP | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | FileSourceScanExec | ColumnarFileSourceScanExec | S | S | S | S | S | S | S | S | S | S | S | | ProjectExec | ColumnarProjectExec | S | S | S | S | S | S | S | S | S | NS | S | | FilterExec | ColumnarFilterExec | S | S | S | S | S | S | S | S | S | NS | S | | ProjectExec+FilterExec | ColumnarConditionProjectExec | S | S | S | S | S | S | S | S | S | NS | S | | ExpandExec | ColumnarExpandExec | S | S | S | S | S | S | S | S | S | NS | S | | HashAggregateExec | ColumnarHashAggregateExec | S | S | S | S | S | S | S | S | S | S | S | | TopNSortExec | ColumnarTopNSortExec | S | S | S | S | S | S | S | S | S | NS | S | | SortExec | ColumnarSortExec | S | S | S | S | S | S | S | S | S | S | S | | BroadcastExchangeExec | ColumnarBroadcastExchangeExec | S | S | S | S | S | S | S | S | S | S | S | | TakeOrderedAndProjectExec | ColumnarTakeOrderedAndProjectExec | S | S | S | S | S | S | S | S | S | S | S | | UnionExec | ColumnarUnionExec | S | S | S | S | S | S | S | S | S | S | S | | ShuffleExchangeExec | ColumnarShuffleExchangeExec | S | S | S | S | S | S | S | S | S | S | S | | BroadcastHashJoinExec | ColumnarBroadcastHashJoinExec | S | S | S | S | S | S | S | S | S | S | S | | SortMergeJoinExec | ColumnarSortMergeJoinExec | S | S | S | S | S | S | S | S | S | S | S | | WindowExec | ColumnarWindowExec | S | S | S | S | S | S | S | S | S | S | S | | ShuffledHashJoinExec | ColumnarShuffledHashJoinExec | S | S | S | S | S | S | S | S | S | S | S | | LocalLimitExec | ColumnarLocalLimitExec | S | S | S | S | S | S | S | S | S | S | S | | GlobalLimitExec | ColumnarGlobalLimitExec | S | S | S | S | S | S | S | S | S | S | S | | CoalesceExec | ColumnarCoalesceExec | S | S | S | S | S | S | S | S | S | S | S | | SubqueryBroadcastExec | OmniColumnarSubqueryBroadcastExec | S | S | S | S | S | S | S | S | S | S | S | | AQEShuffleReadExec | OmniAQEShuffleReadExec | S | S | S | S | S | S | S | S | S | S | S | #### 支持表达式列表 Spark引擎使用OmniOperator算子加速特性支持的表达式/函数情况如表3所示。 **表3 支持表达式列表** | 表达式 | OmniOperator算子加速支持状态 | 函数类型 | OmniOperator算子加速限制描述 | BOOLEAN | INT | LONG | DOUBLE | STRING | CHAR | VARCHAR | DATE | DECIMAL | NULL | SHORT | TIMESTAMP | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | ! | S | Scalar Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | % | S | Scalar Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | \* | S | Scalar Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | + | S | Scalar Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | \- | S | Scalar Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | / | S | Scalar Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | < | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | <= | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | > | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | >= | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | and | S | Scalar Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | any | S | Aggregate Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | avg | S | Aggregate Functions | \- | NA | S | S | S | S | S | S | NA | S | S | S | NA | | between | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | bool\_and | S | Aggregate Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | bool\_or | S | Aggregate Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | case | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | cast | S | Scalar Functions | 详情请参见表4。 | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | | char\_length | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | character\_length | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | count | PS | Aggregate Functions | 仅支持一个入参的情况。 | S | S | S | S | S | S | S | S | S | S | S | S | | count\_if | S | Aggregate Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | current\_catalog | NA\-2 | Scalar Functions | \- | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | | current\_database | NA\-2 | Scalar Functions | \- | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | | current\_date | NA\-2 | Scalar Functions | \- | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | | current\_timezone | NA\-2 | Scalar Functions | \- | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | | current\_user | NA\-2 | Scalar Functions | \- | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | [Blank Cell] | | every | S | Aggregate Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | first | S | Aggregate Functions | \- | S | S | S | S | NS | NS | NS | S | S | NS | S | S | | first\_value | S | Aggregate Functions | \- | S | S | S | S | NS | NS | NS | S | S | NS | S | S | | grouping\_id | S | Aggregate Functions | \- | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | | if | S | Scalar Functions | \- | S | S | S | S | S | S | S | S | S | S | NS | S | | instr | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | isnotnull | S | Scalar Functions | \- | S | S | S | S | S | S | S | S | S | S | NS | S | | isnull | S | Scalar Functions | \- | S | S | S | S | S | S | S | S | S | S | NS | S | | lcase | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | left | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | length | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | lower | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | max | S | Aggregate Functions | \- | S | S | S | S | NS | NS | NS | S | S | NS | S | S | | md5 | PS | Scalar Functions | 入参仅支持String类型的变量。 | NA | NA | NA | NA | S | S | S | NA | NA | S | NA | NA | | mean | S | Aggregate Functions | \- | NA | S | S | S | S | S | S | NA | S | S | S | NA | | min | S | Aggregate Functions | \- | S | S | S | S | NS | NS | NS | S | S | NS | S | S | | mod | S | Scalar Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | not | S | Scalar Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | nullif | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | nvl2 | S | Scalar Functions | \- | S | S | S | S | S | S | S | S | S | S | NS | S | | or | S | Scalar Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | positive | S | Scalar Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | rank | S | Window Functions | 不涉及入参 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | | replace | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | round | S | Scalar Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | row\_number | S | Window Functions | 不涉及入参 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA | | some | S | Aggregate Functions | \- | S | NA | NA | NA | NA | NA | NA | NA | NA | S | NA | NA | | substr | S | Scalar Functions | \- | NA | S | S | S | S | S | S | S | S | S | NS | NS | | substring | S | Scalar Functions | \- | NA | S | S | S | S | S | S | S | S | S | NS | NS | | trunc | S | Scalar Functions | \- | NA | NA | NA | NA | S | S | S | S | NA | S | NA | NS | | ucase | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | upper | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | when | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | != | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | <> | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | = | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | == | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | abs | S | Scalar Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | concat | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | contains | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | decode | PS | Scalar Functions | 仅支持入参数量大于2的情况。 | NS | S | S | S | S | S | S | S | S | S | NS | NS | | endswith | PS | Scalar Functions | 第二个入参仅支持String类型的常量。 | NS | S | S | S | S | S | S | S | S | S | NS | NS | | hash | S | Scalar Functions | \- | S | S | S | S | S | S | S | S | S | S | NS | S | | ifnull | S | Scalar Functions | \- | S | S | S | S | S | S | S | S | S | S | NS | S | | in | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | S | | like | PS | Scalar Functions | 第二个入参仅支持String类型的常量，且该常量中不能包含'\_'以及多个'%'。 | NS | S | S | S | S | S | S | S | S | S | NS | NS | | nvl | S | Scalar Functions | \- | S | S | S | S | S | S | S | S | S | S | NS | S | | regexp | PS | Scalar Functions | 第二个入参必须为String类型的常量'^\\d+$'。 | NS | S | S | S | S | S | S | S | S | S | NS | NS | | regexp\_like | PS | Scalar Functions | 第二个入参必须为String类型的常量'^\\d+$'。 | NS | S | S | S | S | S | S | S | S | S | NS | NS | | regr\_avgx | S | Aggregate Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | regr\_avgy | S | Aggregate Functions | \- | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | right | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | rlike | PS | Scalar Functions | 第二个入参必须为String类型的常量'^\\d+$'。 | NS | S | S | S | S | S | S | S | S | S | NS | NS | | startswith | PS | Scalar Functions | 第二个入参仅支持String类型的常量。 | NS | S | S | S | S | S | S | S | S | S | NS | NS | | sum | S | Aggregate Functions | \- | NA | S | S | S | S | S | S | NA | S | S | S | NA | | to\_date | PS | Scalar Functions | 仅支持一个入参的情况。 | NS | S | S | S | S | S | S | S | S | S | NS | NS | | xxhash64 | S | Scalar Functions | \- | S | S | S | S | S | S | S | S | S | S | NS | S | | \|\| | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | bigint | S | Scalar Functions | \- | NS | S | S | S | S | S | S | NS | S | S | NS | NS | | boolean | S | Scalar Functions | \- | S | NS | NS | NS | NS | NS | NS | NS | NS | S | NS | NS | | date | S | Scalar Functions | \- | NA | NA | NA | NA | S | S | S | S | NA | S | NA | NS | | decimal | S | Scalar Functions | \- | NS | S | S | S | S | S | S | NS | S | S | NS | NS | | double | S | Scalar Functions | \- | NS | S | S | S | S | S | S | NS | S | S | NS | NS | | int | S | Scalar Functions | \- | NS | S | S | S | S | S | S | NS | S | S | NS | NS | | string | S | Scalar Functions | \- | NS | S | S | S | S | S | S | S | S | S | NS | NS | | coalesce | PS | Scalar Functions | 仅支持两个入参的情况。 | S | S | S | S | S | S | S | S | S | S | NS | S | | from\_unixtime | PS | Scalar Functions | 仅支持format为yyyy\-MM\-dd和yyyy\-MM\-dd HH:mm:ss的情况，且时区需要在"GMT+08:00","Asia/Beijing","Asia/Shanghai"之中。 | NA | S | S | S | S | S | S | NA | S | S | NS | NA | | greatest | PS | Scalar Functions | 仅支持两个入参的情况。 | S | S | S | S | S | S | S | NS | S | S | NS | NS | | unix\_timestamp | PS | Scalar Functions | timeExp仅支持String/Date，format仅支持yyyy\-MM\-dd和yyyy\-MM\-dd HH:mm:ss的情况，且时区需要在"GMT+08:00","Asia/Beijing","Asia/Shanghai"之中。 | NA | NA | NA | NA | S | S | S | S | NA | S | NA | NS | **表4 cast表达式支持列表** | 源类型\目标类型 | BOOLEAN | INT | LONG | DOUBLE | STRING | CHAR | VARCHAR | DATE | DECIMAL | SHORT | TIMESTAMP | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BOOLEAN | S | NS | NS | NS | NS | NS | NS | NA | NS | NS | NS | | INT | NS | S | S | S | S | S | S | NA | S | NS | NS | | LONG | NS | S | S | S | S | S | S | NA | S | NS | NS | | DOUBLE | NS | S | S | S | S | S | S | NA | S | NS | NS | | STRING | NS | S | S | S | S | S | S | S | S | NS | NS | | CHAR | NS | S | S | S | S | S | S | S | S | NS | NS | | VARCHAR | NS | S | S | S | S | S | S | S | S | NS | NS | | DATE | NS | NS | NS | NS | S | S | S | S | NS | NS | NS | | DECIMAL | NS | S | S | S | S | S | S | NA | S | NS | NS | | NULL | S | S | S | S | S | S | S | S | S | NS | NS | | SHORT | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | | TIMESTAMP | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS | NS |