介绍Spark 3.3.1引擎在使用OmniOperator算子加速特性时，OmniOperator算子加速特性支持的算子和表达式，以及支持的算子和表达式的数据类型列表。
算子和表达式支持列表中使用符号表示它们的支持状态，符号说明如表1所示。
状态 |
说明 |
|---|---|
S |
表示OmniOperator算子加速支持。 |
PS |
表示OmniOperator算子加速部分支持，存在一些限定条件。 |
NS |
表示OmniOperator算子加速不支持。 |
NA |
表示不涉及，Spark原生也没有此输入场景。 |
NA-2 |
表示Spark上下文函数，Spark内置实现，不涉及OmniOperator算子加速。 |
[Blank Cell] |
表示不适用或需要确认的情况。 |
支持算子列表
Spark引擎使用OmniOperator算子加速特性支持的算子情况如表2所示。
原生算子名称 |
OmniOperator算子加速算子名称 |
BOOLEAN |
INT |
LONG |
DOUBLE |
STRING |
CHAR |
VARCHAR |
DATE |
DECIMAL |
SHORT |
TIMESTAMP |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
FileSourceScanExec |
ColumnarFileSourceScanExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
ProjectExec |
ColumnarProjectExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
FilterExec |
ColumnarFilterExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
ProjectExec+FilterExec |
ColumnarConditionProjectExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
ExpandExec |
ColumnarExpandExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
HashAggregateExec |
ColumnarHashAggregateExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
TopNSortExec |
ColumnarTopNSortExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
SortExec |
ColumnarSortExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
BroadcastExchangeExec |
ColumnarBroadcastExchangeExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
TakeOrderedAndProjectExec |
ColumnarTakeOrderedAndProjectExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
UnionExec |
ColumnarUnionExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
ShuffleExchangeExec |
ColumnarShuffleExchangeExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
BroadcastHashJoinExec |
ColumnarBroadcastHashJoinExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
SortMergeJoinExec |
ColumnarSortMergeJoinExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
WindowExec |
ColumnarWindowExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
ShuffledHashJoinExec |
ColumnarShuffledHashJoinExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
LocalLimitExec |
ColumnarLocalLimitExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
GlobalLimitExec |
ColumnarGlobalLimitExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
CoalesceExec |
ColumnarCoalesceExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
SubqueryBroadcastExec |
OmniColumnarSubqueryBroadcastExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
AQEShuffleReadExec |
OmniAQEShuffleReadExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
支持表达式列表
Spark引擎使用OmniOperator算子加速特性支持的表达式/函数情况如表3所示。
表达式 |
OmniOperator算子加速支持状态 |
函数类型 |
OmniOperator算子加速限制描述 |
BOOLEAN |
INT |
LONG |
DOUBLE |
STRING |
CHAR |
VARCHAR |
DATE |
DECIMAL |
NULL |
SHORT |
TIMESTAMP |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
! |
S |
Scalar Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
% |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
* |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
+ |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
- |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
/ |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
< |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
<= |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
> |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
>= |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
and |
S |
Scalar Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
any |
S |
Aggregate Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
avg |
S |
Aggregate Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
S |
NA |
between |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
bool_and |
S |
Aggregate Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
bool_or |
S |
Aggregate Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
case |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
cast |
S |
Scalar Functions |
详情请参见表4。 |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
char_length |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
character_length |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
count |
PS |
Aggregate Functions |
仅支持一个入参的情况。 |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
count_if |
S |
Aggregate Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
current_catalog |
NA-2 |
Scalar Functions |
- |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
current_database |
NA-2 |
Scalar Functions |
- |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
current_date |
NA-2 |
Scalar Functions |
- |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
current_timezone |
NA-2 |
Scalar Functions |
- |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
current_user |
NA-2 |
Scalar Functions |
- |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
[Blank Cell] |
every |
S |
Aggregate Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
first |
S |
Aggregate Functions |
- |
S |
S |
S |
S |
NS |
NS |
NS |
S |
S |
NS |
S |
S |
first_value |
S |
Aggregate Functions |
- |
S |
S |
S |
S |
NS |
NS |
NS |
S |
S |
NS |
S |
S |
grouping_id |
S |
Aggregate Functions |
- |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
if |
S |
Scalar Functions |
- |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
instr |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
isnotnull |
S |
Scalar Functions |
- |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
isnull |
S |
Scalar Functions |
- |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
lcase |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
left |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
length |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
lower |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
max |
S |
Aggregate Functions |
- |
S |
S |
S |
S |
NS |
NS |
NS |
S |
S |
NS |
S |
S |
md5 |
PS |
Scalar Functions |
入参仅支持String类型的变量。 |
NA |
NA |
NA |
NA |
S |
S |
S |
NA |
NA |
S |
NA |
NA |
mean |
S |
Aggregate Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
S |
NA |
min |
S |
Aggregate Functions |
- |
S |
S |
S |
S |
NS |
NS |
NS |
S |
S |
NS |
S |
S |
mod |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
not |
S |
Scalar Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
nullif |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
nvl2 |
S |
Scalar Functions |
- |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
or |
S |
Scalar Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
positive |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
rank |
S |
Window Functions |
不涉及入参 |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
replace |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
round |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
row_number |
S |
Window Functions |
不涉及入参 |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
some |
S |
Aggregate Functions |
- |
S |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
NA |
S |
NA |
NA |
substr |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
substring |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
trunc |
S |
Scalar Functions |
- |
NA |
NA |
NA |
NA |
S |
S |
S |
S |
NA |
S |
NA |
NS |
ucase |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
upper |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
when |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
!= |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
<> |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
= |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
== |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
abs |
S |
Scalar Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
concat |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
contains |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
decode |
PS |
Scalar Functions |
仅支持入参数量大于2的情况。 |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
endswith |
PS |
Scalar Functions |
第二个入参仅支持String类型的常量。 |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
hash |
S |
Scalar Functions |
- |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
ifnull |
S |
Scalar Functions |
- |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
in |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
like |
PS |
Scalar Functions |
第二个入参仅支持String类型的常量，且该常量中不能包含'_'以及多个'%'。 |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
nvl |
S |
Scalar Functions |
- |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
regexp |
PS |
Scalar Functions |
第二个入参必须为String类型的常量'^\\d+$'。 |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
regexp_like |
PS |
Scalar Functions |
第二个入参必须为String类型的常量'^\\d+$'。 |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
regr_avgx |
S |
Aggregate Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
regr_avgy |
S |
Aggregate Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
right |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
rlike |
PS |
Scalar Functions |
第二个入参必须为String类型的常量'^\\d+$'。 |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
startswith |
PS |
Scalar Functions |
第二个入参仅支持String类型的常量。 |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
sum |
S |
Aggregate Functions |
- |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
S |
NA |
to_date |
PS |
Scalar Functions |
仅支持一个入参的情况。 |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
xxhash64 |
S |
Scalar Functions |
- |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
|| |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
bigint |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
NS |
S |
S |
NS |
NS |
boolean |
S |
Scalar Functions |
- |
S |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
S |
NS |
NS |
date |
S |
Scalar Functions |
- |
NA |
NA |
NA |
NA |
S |
S |
S |
S |
NA |
S |
NA |
NS |
decimal |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
NS |
S |
S |
NS |
NS |
double |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
NS |
S |
S |
NS |
NS |
int |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
NS |
S |
S |
NS |
NS |
string |
S |
Scalar Functions |
- |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
coalesce |
PS |
Scalar Functions |
仅支持两个入参的情况。 |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
from_unixtime |
PS |
Scalar Functions |
仅支持format为yyyy-MM-dd和yyyy-MM-dd HH:mm:ss的情况，且时区需要在"GMT+08:00","Asia/Beijing","Asia/Shanghai"之中。 |
NA |
S |
S |
S |
S |
S |
S |
NA |
S |
S |
NS |
NA |
greatest |
PS |
Scalar Functions |
仅支持两个入参的情况。 |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
S |
NS |
NS |
unix_timestamp |
PS |
Scalar Functions |
timeExp仅支持String/Date，format仅支持yyyy-MM-dd和yyyy-MM-dd HH:mm:ss的情况，且时区需要在"GMT+08:00","Asia/Beijing","Asia/Shanghai"之中。 |
NA |
NA |
NA |
NA |
S |
S |
S |
S |
NA |
S |
NA |
NS |
源类型\目标类型 |
BOOLEAN |
INT |
LONG |
DOUBLE |
STRING |
CHAR |
VARCHAR |
DATE |
DECIMAL |
SHORT |
TIMESTAMP |
|---|---|---|---|---|---|---|---|---|---|---|---|
BOOLEAN |
S |
NS |
NS |
NS |
NS |
NS |
NS |
NA |
NS |
NS |
NS |
INT |
NS |
S |
S |
S |
S |
S |
S |
NA |
S |
NS |
NS |
LONG |
NS |
S |
S |
S |
S |
S |
S |
NA |
S |
NS |
NS |
DOUBLE |
NS |
S |
S |
S |
S |
S |
S |
NA |
S |
NS |
NS |
STRING |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
CHAR |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
VARCHAR |
NS |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
DATE |
NS |
NS |
NS |
NS |
S |
S |
S |
S |
NS |
NS |
NS |
DECIMAL |
NS |
S |
S |
S |
S |
S |
S |
NA |
S |
NS |
NS |
NULL |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
NS |
SHORT |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
TIMESTAMP |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
NS |
NS |