Error Reported When Submitting an Algorithm Task Through spark-submit
Symptom
When a Random Forest, Decision Tree, or GBDT algorithm task is submitted through spark-submit, and spark.driver.userClassPathFirst or spark.executor.userClassPathFirst is set to true, the algorithm is terminated unexpectedly and a ClassCastException or loader constraint violation error is reported.
Key Process and Cause Analysis
The two parameters are experimental parameters, which change the loading sequence of JAR files of Spark tasks. As a result, class conflicts occur.
Conclusion and Solution
Remove the spark.driver.userClassPathFirst=true or spark.executor.userClassPathFirst=true configuration when using spark-submit to submit tasks.
Parent topic: Troubleshooting