Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Configuring and Setting Up the Environment

Environment Configuration

Host Name

Number of Drives

Memory

Network

server1

System drive: 2 x 600 GB SAS drive configured in RAID 1

Data drive: 12 x 1.2 TB SAS drive configured in RAID 0

256 GB (8 x 32 GB)

10GE

agent1

agent2

agent3

Installing Maven

  1. Download the installation package and install Maven to a directory (for example, /opt/tools/installed/).
    1
2
3
    		 
    wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz
tar -zxf apache-maven-3.5.4-bin.tar.gz
mv apache-maven-3.5.4 /opt/tools/installed/
  2. Modify the Maven environment variables.
    1. Open the configuration file. 
      1
      		 
      vi /etc/profile
    2. Press i to enter the insert mode and add the following content to the end of the /etc/profile file. 
      1
2
      		 
      export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4
export PATH=$MAVEN_HOME/bin:$PATH
    3. Press Esc, type :wq!, and press Enter to save the file and exit.
  3. Make the modified environment variables take effect.
    1
    		 
    source /etc/profile
  4. Check whether Maven is successfully installed.
    1
    		 
    mvn -v

    The installation is successful if information similar to the following is displayed:

  5. Modify the local repository path and remote repository in the Maven configuration file.

    Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml

    The default local repository directory is ~/.m2/. If you want to change the directory to a specified one, modify the localRepository tag. You do not need to modify this parameter unless otherwise specified.

    Add the following content to the <mirrors> tag to configure the remote repository (change the repository to the Maven repository that you have built. If the Maven repository does not exist, configure it based on the following example):

    1
2
3
4
5
6
    		 
    <mirror>
  <id>huaweimaven</id>
  <name>huawei maven</name>
  <url>https://mirrors.huaweicloud.com/repository/maven/</url>
  <mirrorOf>central</mirrorOf>
</mirror>

    If the compilation environment cannot access the Internet, add the following proxy configuration to the settings.xml file:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    		 
    <proxies>
  <proxy>
    <id>optional</id>
    <active>true</active>
    <protocol>http</protocol>
    <username>User name</username>
    <password>Password</password>
    <host>Proxy server URL</host>
    <port>Proxy server port</port>
    <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
  </proxy>
</proxies>

Installing Lein

  1. Go to the /usr/local/bin directory.
    1
    		 
    cd /usr/local/bin
  2. Download the lein file.
    1
    		 
    wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
  3. Assign the execution permission on the file.
    1
    		 
    chmod 755 lein

Setting Up the Test Environment

Yahoo Streaming Benchmark simulates a simple ad scenario to compare stream computing frameworks. The specific scenario is as follows: The clickstream of the ad consumed by Kafka is associated with the campaign information of the ad in Redis, and then time window aggregation counting is performed.

Yahoo Streaming Benchmark comes with Zookeeper, Kafka, Redis, and Flink, which can be directly used for testing.

If you need to use Flink and Kafka of a customized version, download the installation packages of the specified versions.