Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Configuring the SSL Encryption Certificate

To ensure secure communication between OmniAdvisor and the Spark history server, as well as between OmniAdvisor and PostgreSQL, it is recommended to enable SSL encryption. By default, SSL encryption is enabled for communication. However, you still need to manually set certificate parameters.

Set certificate parameters as follows:

  1. Generate an SSL certificate by following instructions in the PostgreSQL official document.
  2. Generate a Spark history server encryption certificate.

    The following is an example script for generating a Spark history server encryption certificate. Replace the example values of SERVER_IP, CLIENT_IP, and path variables with the actual values.

    #!/bin/bash
# --------------------------------------------------
# Script for generating the Spark Mutual TLS (mTLS) two-way authentication certificate
# The certificates used by the Spark client and Spark history server are issued by the same self-signed CA.
# --------------------------------------------------
set -euo pipefail
# Set the variables.
HOME_DIR="/home/omniadvisor_test/ssl"
CA_DIR="$HOME_DIR/spark-tls-ca"
CLIENT_DIR="$HOME_DIR/spark-client"
SERVER_DIR="$HOME_DIR/spark-history-server"
PKCS12_PASS="123456"
SERVER_IP="6.6.6.6"
CLIENT_IP="6.6.6.6"
CA_KEY="$CA_DIR/ca.key"
CA_CERT="$CA_DIR/ca.crt"
CLIENT_KEY="$CLIENT_DIR/spark-client.key"
CLIENT_CSR="$CLIENT_DIR/spark-client.csr"
CLIENT_CERT="$CLIENT_DIR/spark-client.crt"
CLIENT_P12="$CLIENT_DIR/spark-history-client.p12"
SERVER_KEY="$SERVER_DIR/spark-history-server.key"
SERVER_CSR="$SERVER_DIR/spark-history-server.csr"
SERVER_CERT="$SERVER_DIR/spark-history-server.crt"
SERVER_P12="$SERVER_DIR/spark-history-server.p12"
# Create directories.
mkdir -p "$CA_DIR" "$CLIENT_DIR" "$SERVER_DIR"
echo "[*] Generating the CA key and certificate..."
# Generate a CA private key.
openssl genrsa -out "$CA_KEY" 2048
# 5. Generate a SAN extension file.
cat > "$CA_DIR/san.ext" <<EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = $SERVER_IP
EOF
# Generate a CA self-signed certificate (valid for 10 years).
openssl req -x509 -new -nodes -key "$CA_KEY" \
  -sha256 -days 365 \
  -out "$CA_CERT" \
  -subj "/C=CN/ST=Beijing/L=Beijing/O=SparkInternal/OU=Dev/CN=$SERVER_IP"
echo "[✓] CA generated: $CA_KEY, $CA_CERT"
# ======================
# Generate a certificate for the client.
# ======================
echo "[*] Generating a key and a certificate for the Spark client..."
# Generate a client private key.
openssl genrsa -out "$CLIENT_KEY" 2048
# Generate a client CSR.
openssl req -new -key "$CLIENT_KEY" \
  -out "$CLIENT_CSR" \
  -subj "/C=CN/ST=Beijing/L=Beijing/O=SparkInternal/OU=Client/CN=$CLIENT_IP"
# Use the CA to issue a client certificate (valid for one year).
openssl x509 -req -in "$CLIENT_CSR" \
  -CA "$CA_CERT" -CAkey "$CA_KEY" -CAcreateserial \
  -out "$CLIENT_CERT" -days 365 -sha256 \
  -extfile "$CA_DIR/san.ext"
# Export the file in .p12 format.
openssl pkcs12 -export \
  -in $CLIENT_CERT \
  -inkey $CLIENT_KEY \
  -out $CLIENT_P12 \
  -name myserver \
  -password pass:$PKCS12_PASS
echo "[✓] Spark client certificate generated: $CLIENT_KEY, $CLIENT_CERT"
# ======================
# Generate a certificate for the Spark history server.
# ======================
echo "[*] Generating a key and a certificate for the Spark history server..."
# Generate a server private key.
openssl genrsa -out "$SERVER_KEY" 2048
# Generate a server CSR.
openssl req -new -key "$SERVER_KEY" \
  -out "$SERVER_CSR" \
  -subj "/C=CN/ST=Beijing/L=Beijing/O=SparkInternal/OU=Server/CN=$SERVER_IP"
# Use the CA to issue a server certificate (valid for one year).
openssl x509 -req -in "$SERVER_CSR" \
  -CA "$CA_CERT" -CAkey "$CA_KEY" -CAcreateserial \
  -out "$SERVER_CERT" -days 365 -sha256 \
  -extfile "$CA_DIR/san.ext"
# Export the file in .p12 format.
openssl pkcs12 -export \
  -in $SERVER_CERT \
  -inkey $SERVER_KEY \
  -out $SERVER_P12 \
  -name myserver \
  -password pass:$PKCS12_PASS
echo "[✓] Spark history server certificate generated: $SERVER_KEY, $SERVER_CERT"
# ======================
# Output a summary.
# ======================
echo ""
echo "=================================================="
echo " Spark mTLS two-way authentication certificate generated"
echo "All files are saved in the subfolders of the current directory:"
echo ""
echo "CA (used to verify the certificates of both parties):"
echo "  - $CA_CERT     : CA public key certificate (needs to be trusted by the client and server)"
echo "  - $CA_KEY      : CA private key (Keep it secure.)"
echo ""
echo "Spark Client:"
echo "  - $CLIENT_KEY  : Client private key"
echo "  - $CLIENT_CERT : Client certificate (issued by the CA)"
echo ""
echo "Spark History Server:"
echo "  - $SERVER_KEY  : Server private key"
echo "  - $SERVER_CERT : Server certificate (issued by the CA)"
echo ""
echo " Two-way authentication principle:"
echo "  - The client uses $CLIENT_CERT to prove its identity to the server."
echo "  - The server uses $SERVER_CERT to prove its identity to the client."
echo "  - Both parties trust $CA_CERT (that is, CA is the root of trust)."
echo ""
echo " Next: Configure these certificates in the SSL configuration of the Spark client and Spark history server."
echo "    Including: spark.ssl.*, spark.history.ui.ssl.*, keyStore, trustStore"
echo "=================================================="
  3. Modify the common_config.ini configuration file.
    1. Go to the configuration file directory and open the configuration file.
      cd $OMNIADVISOR_HOME/omniruntime-omniadvisor-2.0.0/config
vi common_config.ini
    2. Press i to enter the insert mode and modify the following configurations as required:
      [database]
# SSL mode of the backend PostgreSQL database
postgresql.database.sslmode=verify-full
# Path to the server CA root certificate of the backend PostgreSQL database
postgresql.database.sslrootcert=/path/to/pg-sql/ca.crt
# Path to the client root certificate of the backend PostgreSQL database
postgresql.database.sslcert=/path/to/pg-sql/client_postgres.crt
# Path to the client private key of the backend PostgreSQL database
postgresql.database.sslkey=/path/to/pg-sql/client_postgres.key
# Client private key password of the backend PostgreSQL database (If there is no password, leave this parameter blank.)
postgresql.database.sslpassword=

[spark]
# URL of the Spark history server (used only in Rest mode)
spark.history.rest.url=https://<server1>:18480
# SSL mutual verification switch for the Spark history server (enabled by default; mandatory and must not be empty)
spark.history.sslverify = True
# Files related to the SSL mutual verification certificate of the Spark history server (valid only when sslverify is set to True)
spark.history.sslrootca =/path/to/spark-tls-ca/ca.crt
spark.history.sslcrt =/path/to/spark-client/spark-client.crt
spark.history.sslkey =/path/to/spark-client/spark-client.key
    3. Press Esc, type :wq!, and press Enter to save the file and exit.
  4. Configure the Spark history server.
    1. Open the $SPARK_HOME/conf/spark-defaults.conf configuration file.
      vi $SPARK_HOME/conf/spark-defaults.conf
    2. Press i to enter the insert mode and add the following TLS configuration. Set spark.ssl.keyStore, spark.ssl.keyStorePassword, spark.ssl.keyPassword, spark.ssl.trustStore, and spark.ssl.trustStorePassword to their corresponding actual values.
      spark.ssl.historyServer.enabled         true
spark.ssl.protocol                      TLSv1.2

spark.ssl.keyStoreType                  PKCS12
spark.ssl.keyStore                      /path/to/spark-history-server.p12
spark.ssl.keyStorePassword              your_password
spark.ssl.keyPassword                   your_password
spark.ssl.trustStore            /path/to/spark-history-client.p12
spark.ssl.trustStorePassword    your_password
spark.ssl.needClientAuth        true
    3. Press Esc, type :wq!, and press Enter to save the file and exit.
    4. Restart the Spark history server.

      In the sbin directory of Spark, stop and start the history server.

      # Stop the Spark history server.
sh $SPARK_HOME/sbin/stop-history-server.sh
# Start the Spark history server.
sh $SPARK_HOME/sbin/start-history-server.sh
Parent topic: Using the Feature