Network Planning
Coupled storage and compute network architecture is recommended for the subfeatures. Storage nodes and compute nodes are shared to maximize the computing acceleration effect in big data scenarios.
ESS Mode
In ESS mode, OmniShuffle uses coupled storage and compute network architecture, which consists of one management node and three compute nodes.
For example, if the storage nodes are HDFS nodes:
- The management node is server, which manages tasks.
- The compute nodes are agent01, agent02, and agent03, which run the OmniShuffle query engine service and store datasets.
A server can function as the management node and a compute node at the same time. In single-node mode, operations performed on the management node or compute node mentioned in the following sections are performed on the same node. Figure 1 shows the networking diagram.
RSS Mode
- OmniShuffle RSS brings significant performance improvement in a Spark task only when a large amount of data is shuffled.
- OmniShuffle RSS uses decoupled storage and compute architecture, which requires high cluster bandwidth. If the cluster bandwidth is lower than 50GE, it may become a performance bottleneck in processing a large volume of data.
- OmniShuffle RSS stores shuffle data in an independent RSS cluster. The number of RSS nodes is much smaller than that of compute nodes. NVMe SSDs are recommended for executing multiple tasks at the same time because shuffle data needs to be flushed to drives. If HDDs are used instead for a large volume of data, drive I/Os may cause a performance bottleneck.
In RSS mode, OmniShuffle uses decoupled storage and compute architecture, which consists of one management node, six compute nodes, and two storage nodes.
- The management node is server, which manages tasks.
- The compute nodes are agent01, agent02, agent03, agent04, agent05, and agent06, which run the OmniShuffle query engine service.
- The storage nodes are RSS01 and RSS02, which store datasets during shuffle.
Figure 2 shows the networking diagram.
