Saving Run Logs to a Local PC

By default, the logs generated when an algorithm is running are displayed on the client and are not stored in files. You can import the customized log4j.properties file to save the logs to your local PC.

Create a log4j.properties file and configure it as follows:

# Set the global default log level and output link.
log4j.rootLogger = INFO,ROLLFILE

# Define the output link.
log4j.addivity.org.apache=true
log4j.appender.ROLLFILE=org.apache.log4j.RollingFileAppender  
log4j.appender.ROLLFILE.Threshold=INFO
# Log output path, which can be customized, for example, /home/test/boostkit/.
log4j.appender.ROLLFILE.File=/home/test/boostkit/spark_algorithm.log  
log4j.appender.ROLLFILE.Append=true
# Set the size of each log file to be rolled back.
log4j.appender.ROLLFILE.MaxFileSize=200MB 
# Set the maximum number of backup files for log rollback. The maximum value is 11.
log4j.appender.ROLLFILE.MaxBackupIndex=11
log4j.appender.ROLLFILE.layout=org.apache.log4j.PatternLayout

The log4j.appender.ROLLFILE.MaxFileSize and log4j.appender.ROLLFILE.MaxBackupIndex parameters in the log4j.properties file can be adjusted based on the log volume and security audit requirements.

Place the customized log4j.properties file in the same directory (for example, /home/test/boostkit/) as the shell script used for submitting tasks.
Modify the shell script for submitting tasks and add the following configuration to spark-submit:
1
--driver-java-options "-Dlog4j.configuration=file:./log4j.properties" \

Parent topic: Log Management