Architecture
The OmniStream Flink Native feature uses native code (C/C++) to reconstruct the logic of Flink SQL and DataStream operators, improving query performance.
- For SQL, OmniStream uses C++ and vectorized instructions to implement operators, leveraging vectorization to enhance SQL computing performance.
- For DataStream, OmniStream uses C++ and vectorized instructions to implement operators, fully leveraging the performance advantages of native code to improve performance in DataStream scenarios.
SQL
OmniStream uses an architecture consisting of the Java Adapter layer and the CPP Core layer.
- The Java Adapter layer is implemented in Java and is responsible for generating native execution plans and falling back to the Java runtime in unsupported scenarios.
- The CPP Core layer is implemented in C++ and is responsible for implementing operator logic and data transmission.
An SQL query submitted through SQL or Table API is parsed, and an execution plan is then generated. The Java Adapter layer obtains the execution plan, initializes related tasks in the CPP, and generates an operator chain. After the initialization process is complete, the task runs to read data from the source. After a series of operator processing operations, Sink outputs the result.
Figure 1 shows the architecture of OmniStream Flink SQL Native.
DataStream
After receiving an input, the DataStream API parses it into an execution plan. The Java adapter layer parses the plan, initializes related tasks on the C++ side, and builds the corresponding operator chain. After the initialization process is complete, the task runs to read data from the source. After a series of operator processing operations, Sink outputs the result.
Figure 2 shows the architecture of OmniStream Flink DataStream Native.
