Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

PySpark Task Submission

  • In the following command, mlpy_env.zip is used as an example. You can change the value as required.
  • The task script /opt/test_spark.py is an example and can be replaced.
  1. Upload the Python virtual environment package mlpy_env.zip to the Python cluster (for example, the /opt directory).
    1
    		 
    /opt/mlpy_env.zip
  2. Submit a PySpark task to a mixed-deployment cluster.
    1
    		 
    PYSPARK_PYTHON=./ANACONDA/mlpy_env/bin/python spark-submit --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./ANACONDA/mlpy_env/bin/python --conf spark.executorEnv.PYSPARK_PYTHON=./ANACONDA/mlpy_env/bin/python --master yarn-cluster --archives /opt/mlpy_env.zip#ANACONDA /opt/test_spark.py
  3. Check the task execution result.

    The distribution of executors is as follows.

    In the preceding figure, agent1 is an x86 server, and agent2 is an Arm server.