Network Planning

A storage-compute coupled networking architecture is recommended for the subfeatures. Storage nodes and compute nodes are shared to maximize the computing acceleration effect in big data scenarios. If OmniShuffle and OmniOperator are used together, you are also advised to plan the network based on this section.

The OmniShuffle network environment consists of four servers, which are one management node and three compute nodes. A coupled storage and compute network is used. In the following, we will be using HDFS as the storage node for illustration:

• The management node is server1 for managing tasks.
• The compute nodes are agent1, agent2, and agent3, which are used to run the OmniShuffle query engine service and store datasets.

A server can function as a management node and a compute node at the same time. In single-node mode, operations performed on the management node or compute node mentioned in the following sections are performed on the same node. Figure 1 shows the networking diagram.