Software Architecture

OmniData supports access to popular data types such as ORC, Parquet, and TXT. It allows Hive and Spark to push down the Filter, Aggregation, and Limit operators to the CPU on the storage node to implement near-data computing, reducing transmission of invalid data on the network and improving big data computing performance. This feature interconnects with HAF and the distributed storage system Ceph or HDFS. OmniData consists of four parts, as shown in Figure 1.

Figure 1 Software architecture of OmniData

• OmniData Client is an open source component that provides plugins for different engines. After annotations are added to functions to be pushed down using the annotation and compilation plugins provided by HAF, HAF automatically pushes tasks to the OmniData Server of the offload node, facilitating task execution.
• Host Runtime is a library deployed on the compute node (host node) to provide the task offload capability and push tasks to Target Runtime.
• Target Runtime is a library deployed on the storage node (offload node) and provides the task execution capability and execute OmniData Server jobs.
• OmniData Server handles tasks pushed down by Host Runtime.