文档
我要评分
获取效率
正确性
完整性
易理解
提交关闭

Typical Configuration

Table 1 describes the configuration of each component in real-time stream processing scenarios.

Table 1 Typical configuration in big data real-time stream processing scenarios

Node Type

Typical Configuration

Node Quantity

Quantity Calculation

Management node

Two-socket rack server, two Huawei Kunpeng 916 or Kunpeng 920 processors, 128 GB or larger memory, six 600 GB or larger 2.5-inch SAS drives, 1 GB LSI RAID 0/1 card (more than three RAID 1 arrays are supported), two 10GE network ports, two GE network ports (the two network ports are bonded and connected to two access switches), and independent power supply

2

Active/standby architecture: one active node + one standby node

Controller node

Two-socket rack server, two Huawei Kunpeng 916 or Kunpeng 920 processors, 256 GB or larger memory, ten 600 GB or larger 2.5-inch SAS drives, 1 GB LSI RAID 0/1 card (more than five RAID 1 arrays are supported), two 10GE network ports, two GE network ports (the two network ports are bonded and connected to two access switches), and independent power supply

3/5/9/11
  • A cluster with 30 to 100 nodes: three controller nodes
  • A cluster with 100 to 500 nodes: five controller nodes
  • A cluster with 500 to 2000 nodes: nine controller nodes
  • A cluster with 2000 to 5000 nodes: 11 controller nodes

Management and controller nodes (hybrid deployment)

Two-socket rack server, two Huawei Kunpeng 916 or Kunpeng 920 processors, 256 GB or larger memory, twelve 600 GB or larger 2.5-inch SAS drives, 1 GB LSI RAID 0/1 card (more than six RAID 1 arrays are supported), two 10GE network ports, two GE network ports (the two network ports are bonded and connected to two access switches), and independent power supply

3

A cluster with 3 to 30 nodes

Kafka

Two-socket server, two Huawei Kunpeng 916 or Kunpeng 920 processors, 256 GB or larger memory, two 600 GB 2.5-inch SAS drives, twenty-four 1.2 TB or larger 2.5-inch SAS drives, 1 GB LSI RAID 0/1 card (more than one RAID 1 array is supported), two 10GE network ports, and two GE network ports (the two network ports are bonded and connected to two access switches), and independent power supply

Calculate based on the throughput or storage data volume.
  • Calculate by throughput: X (total throughput, MB/s)/100 (maximum producer throughput per node, MB/s)/0.85 (reserved ratio)
  • Calculate based on the storage capacity: X (total throughput, MB/s) x 3600 x 24 x D (quantity of days, 7 by default) x 2 (quantity of copies)/1024/1024/(24 x 1.2)/0.85 (reservation ratio)
  • Use the larger value. At least two devices must be configured.

Flink/Storm

Two-socket rack server, two Huawei Kunpeng 916 or Kunpeng 920 processors, 256 GB or larger memory, four 600 GB or larger 2.5-inch SAS drives, 1 GB LSI RAID 0/1 controller card (more than one RAID 1 arrays are supported), two 10GE network ports, two GE network ports (the two network ports are bonded and connected to two access switches), and independent power supply

Calculated based on the computing volume
  • Flink: X (total throughput, MB/s)/30 (throughput per node, MB/s)/0.85 (reservation ratio)
  • Spark Streaming: X (total throughput, MB/s)/10 (throughput per node, MB/s)/0.85 (reservation ratio)

Redis

Two-socket rack server, two Huawei Kunpeng 916 or Kunpeng 920 processors, 512 GB or larger memory, ten 600 GB or larger 2.5-inch SAS drives, 1 GB LSI RAID 0/1 card (more than one RAID 1 arrays are supported), two 10GE network ports, two GE network ports (the two network ports are bonded and connected to two access switches), and independent power supply

Calculated based on the data volume

M (total data volume, GB) x 2 (two copies)/512 (memory of a single node, GB)/0.85 (reservation ratio)