Supported Operators and Expressions
This section describes the operators, expressions, and data types supported by OmniOperator when Spark uses OmniOperator on Gluten.
Table 2 and Supported Expressions describe the operators, expressions, and functions supported by OmniOperator on Gluten. Symbols are used to indicate whether the operators and expressions are supported. For details about the meanings of the symbols, see Table 1.
- Operator List and Supported Expressions list only the data types supported by OmniOperator. Other data types (BYTE/FLOAT/BINARY/ARRAY/MAP/STRUCT/CALENDAR/UDT) are not supported.
- If you use operators and expressions that are not supported by OmniOperator, the execution plan will be rolled back to open source execution, which deteriorates the performance.
Status |
Description |
|---|---|
S |
Indicates that the operator or expression is supported. |
PS |
Indicates that the operator or expression is partially supported, with some restrictions. For details about the restrictions, see Constraints. |
NS |
Indicates that the operator or expression is not supported. |
NA |
Indicates that the operator or expression is not involved. This scenario does not exist in open source Spark. |
NA-2 |
Indicates a context function implemented based on open source Spark, which does not involve using OmniOperator. |
[Blank Cell] |
Indicates a scenario that is irrelevant or needs to be confirmed. |
Operator List
Table 2 lists the operators supported by OmniOperator for the Spark engine.
Open Source Operator |
OmniOperator Operator |
BOOLEAN |
INT |
LONG |
DOUBLE |
STRING |
CHAR |
VARCHAR |
DATE |
DECIMAL |
SHORT |
TIMESTAMP |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
FileSourceScanExec |
FileSourceScanExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
ProjectExec |
ProjectExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
FilterExec |
OmniFilterExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
ExpandExec |
ExpandExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
HashAggregateExec |
OmniHashAggregateExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
TopNSortExec |
OmniTopNSortTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
NS |
S |
SortExec |
SortExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
BroadcastExchangeExec |
ColumnarBroadcastExchangeExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
TakeOrderedAndProjectExec |
TakeOrderedAndProjectExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
UnionExec |
UnionExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
ShuffleExchangeExec |
OmniColumnarShuffleExchangeExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
BroadcastHashJoinExec |
OmniBroadcastHashJoinExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
SortMergeJoinExec |
OmniSortMergeJoinExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
WindowExec |
WindowExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
ShuffledHashJoinExec |
OmniShuffledHashJoinExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
LocalLimitExec |
ColumnarLocalLimitExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
GlobalLimitExec |
LimitExecTransformer |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
CoalesceExec |
ColumnarCoalesceExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
SubqueryBroadcastExec |
ColumnarSubqueryBroadcastExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
AQEShuffleReadExec |
OmniAQEShuffleReadExec |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |
S |