OmniData

OmniData supports access to popular data types such as ORC and Parquet. It allows Spark to push down the Filter, Aggregation, and Limit operators to CPUs on a storage node to implement near-data computing, reducing transmission of invalid data on the network and improving big data computing performance. This feature interconnects with HAF and the distributed storage system Ceph or HDFS, as shown in Figure 1.

Figure 1 Software architecture of OmniData

• The OmniData Client is an open source component that provides plugins for different engines. After annotations are added to functions to be pushed down using the annotation and compilation plugins provided by HAF, HAF automatically pushes tasks to the OmniData Server of the offload node, facilitating task execution.
• The Host Runtime is a library installed on the compute node to provide the task offload capability and push tasks to Target Runtime.
• The Target Runtime is a library installed on the storage node (offload node) to provide the task execution capability and execute OmniData Server jobs.
• The OmniData Server handles tasks pushed down by the Host Runtime.