Installing Spark

Planning the Cluster Environment

The environment planned in this section consists of seven servers, including one task submission node, three compute nodes, and three storage nodes. In the big data cluster, the Spark driver functions as the task submission node, and the compute nodes are agent1, agent2, and agent3. The storage nodes are ceph1, ceph2, and ceph3. See Figure 1.

Figure 1 Environment configuration

Table 1 lists the hardware environment of the cluster.

Installing Spark 3.1.1

During the installation, select the /opt/boostkit directory as the software installation directory and place all JAR packages on which Spark compilation depends in this directory, as shown in Table 2.

The aws-java-sdk-bundle-1.11.375.jar and hdfs-ceph-3.2.0.jar packages need to be added in the Ceph environment. The HDFS environment does not require these packages. The other packages listed in the previous table can be obtained by running the spark_build.sh script on Gitee.

1. Create an /opt/boostkit directory.

   1	mkdir -p /opt/boostkit

2. On the task submission node (Spark driver), upload the boostkit-omnidata-client-1.5.0-aarch64.jar and boostkit-omnidata-common-1.5.0-aarch64.jar files (contained in BoostKit-omnidata_1.5.0.zip\BoostKit-omnidata_1.5.0.tar.gz) obtained from Obtaining Software to the /opt/boostkit directory.

   1 2	cp boostkit-omnidata-client-1.5.0-aarch64.jar /opt/boostkit cp boostkit-omnidata-common-1.5.0-aarch64.jar /opt/boostkit

3. Upload the haf-1.3.0.jar file (contained in BoostKit-haf_1.3.0.zip\haf-1.3.0.tar.gz\haf-host-1.3.0.tar.gz\lib\jar) obtained in Obtaining Software to the /opt/boostkit directory.

   1	cp haf-1.3.0.jar /opt/boostkit

4. Upload hdfs-ceph-3.2.0.jar obtained in Obtaining Software and aws-java-sdk-bundle-1.11.375.jar in boostkit-omnidata-server-1.5.0-aarch64-lib.zip to the /opt/boostkit directory. (If the HDFS storage system is used, skip this step.)

   1 2	cp hdfs-ceph-3.2.0.jar /opt/boostkit cp aws-java-sdk-bundle-1.11.375.jar /opt/boostkit

5. Use an FTP tool to upload the boostkit-omnidata-spark-sql_2.12-3.1.1-1.5.0-aarch64.zip package obtained in Obtaining Software to the installation environment and decompress the package.

   1	unzip boostkit-omnidata-spark-sql_2.12-3.1.1-1.5.0-aarch64.zip

6. Copy the JAR packages in the boostkit-omnidata-spark-sql_2.12-3.1.1-1.5.0-aarch64.zip package to the /opt/boostkit directory.

   1 2	cd boostkit-omnidata-spark-sql_2.12-3.1.1-1.5.0-aarch64 cp *.jar /opt/boostkit

   

   If you need to manually compile boostkit-omnidata-spark-sql_2.12-3.1.1-1.5.0.jar, compile it based on README.md.

7. Add new OmniData parameters to the Spark configuration file ($SPARK_HOME/conf/spark-defaults.conf).

   Replace $SPARK_HOME with /usr/local/spark.

   1. Open the Spark configuration file.

      1	vi /usr/local/spark/conf/spark-defaults.conf

   2. Press i to go to the insert mode and add the following parameters to spark-defaults.conf:

      1 2 3 4 5 6 7 8 9 10 11 12 13

      spark.sql.cbo.enabled true spark.sql.cbo.planStats.enabled true spark.sql.ndp.enabled true spark.sql.ndp.filter.selectivity.enable true spark.sql.ndp.filter.selectivity 0.5 spark.sql.ndp.alive.omnidata 3 spark.sql.ndp.table.size.threshold 10 spark.sql.ndp.zookeeper.address agent1:2181,agent2:2181,agent3:2181 spark.sql.ndp.zookeeper.path /sdi/status spark.sql.ndp.zookeeper.timeout 15000 spark.driver.extraLibraryPath /home/omm/omnidata-install/haf-host/lib spark.executor.extraLibraryPath /home/omm/omnidata-install/haf-host/lib spark.executorEnv.HAF_CONFIG_PATH /home/omm/omnidata-install/haf-host/etc/

      You can also run the set command to set the preceding parameters in spark-sql.

      Table 3 lists the new OmniData parameters.

      Table 3 OmniData parameters

      Parameter	Recommended Value	Description
      spark.sql.cbo.enabled	true	Indicates whether to enable CBO optimization. If this parameter is set to true, CBO is enabled to estimate the execution plan for statistics collection.
      spark.sql.cbo.planStats.enabled	true	If this parameter is set to true, the logical plan obtains row and column statistics from the catalog.
      spark.sql.ndp.enabled	true	Indicates whether to enable OmniData.
      spark.sql.ndp.filter.selectivity.enable	true	Indicates whether to enable filter selectivity to determine whether to perform operator pushdown.
      spark.sql.ndp.filter.selectivity	0.5	If the actual filter selectivity is less than the value of this parameter, operator pushdown is performed. A smaller value indicates a smaller amount of data to be filtered. The default value is 0.5 and the type is double. This parameter can be set after spark.sql.ndp.filter.selectivity.enable is set to true. If forcible pushdown is required, set this parameter to 1.0.
      spark.sql.ndp.table.size.threshold	10240	Table size threshold for operator pushdown. The default value is 10240, in bytes. When the actual table size is larger than the value, operator pushdown is performed.
      spark.sql.ndp.alive.omnidata	3	Number of OmniData servers in the cluster.
      spark.sql.ndp.zookeeper.address	agent1:2181,agent2:2181,agent3:2181	IP address for connecting to ZooKeeper.
      spark.sql.ndp.zookeeper.path	/sdi/status	ZooKeeper directory for storing pushed-down resource information.
      spark.sql.ndp.zookeeper.timeout	15000	ZooKeeper timeout duration, in milliseconds.
      spark.driver.extraLibraryPath	/home/omm/omnidata-install/haf-host/lib	Path of library files on which the Spark driver depends.
      spark.executor.extraLibraryPath	/home/omm/omnidata-install/haf-host/lib	Path of library files on which the Spark executor depends.
      spark.executorEnv.HAF_CONFIG_PATH	/home/omm/omnidata-install/haf-host/etc/	Path of the configuration file for enabling HAF.

   3. Press Esc, type :wq!, and press Enter to save the file and exit.