Cluster Environment

A cluster consists of one client, one controller node, and three compute nodes. Figure 1 shows the networking diagram. The controller node functions as the server, and the compute nodes are agent1, agent2, and agent3 of the big data cluster. In POC test scenarios, the client can be deployed on the controller node.

Figure 1 Networking diagram

Cluster Hardware

Table 1 lists the hardware configurations in the cluster (controller node and compute nodes).

**Table 1** Hardware configurations
Item	Requirement
Server	TaiShan server
Processor	Huawei Kunpeng 920 processor
Memory size	384 GB (12 x 32 GB)
Memory frequency	2666MHz
NIC	10GE for the service network and GE for the management network
Drive	System drive: 1 x RAID 0 (1 x 1.2 TB SAS HDD) Data drive: 12 x RAID 0 (1 x 4 TB SATA HDD)
RAID controller card	LSI SAS3508

Cluster Software

Table 2 lists the required software versions.

**Table 2** Recommended software configurations in the cluster environment
Item	Requirement
OS	CentOS 7.6
JDK	OpenJDK 1.8.0_252
HDP	3.1.0
Hive	3.1.0
Zookeeper	3.4.9
Hadoop	3.1.1
Spark	2.3.2

For details about cluster deployment, see the Spark Cluster Deployment Guide (CentOS 7.6 & openEuler 20.03). The Spark deployment mode is Spark on Yarn.
The Kunpeng algorithm library supports only HDP 3.1.0 Spark 2.3.2. Other platforms have not been verified.

Parent topic: Installing Software