Classification and Regression

Scenarios

Classification and regression analysis is a predictive modeling technology that explores the relationship between labels and features, where labels may be considered as dependent variables, and features as independent variables. Classification and regression algorithms are usually applied in predictive analysis and modeling regression.

Specifically, algorithms such as Linear Regression and Logistic Regression are used for credit risk analysis of Internet finance P2P services and traffic flow prediction for road networks; SVM used for rice forecast for the international carbon financial market and traffic flow prediction; and GBDT and XGBoost used for debt risk rating and warning and travel mode recommendation.

Regression algorithms use multiple iterations to converge approximately to label variables for model training. The Kunpeng BoostKit big data machine learning algorithm library optimizes iteration algorithms and fully exploits the high-concurrency capabilities of Kunpeng processors to reduce the number of iterations during the training process, improving the algorithm performance by multiple times.

Principles

• GBDT

  Gradient boosting decision tree (GBDT) is a popular decision tree–based ensemble algorithm used for classification and regression tasks. It iteratively trains decision trees to minimize a loss function. Spark GBDT enables binary classification and regression, supports continuous features and categorical features, and uses distributed computing for training and inference in big data scenarios.

• Random Forest

  The Random Forest algorithm trains multiple decision trees simultaneously to obtain a classification model or regression model based on given sample data that includes feature vectors and label values. The output model can predict the label value with the highest probability after the feature vectors are input.

• SVM

  Support vector machine (SVM) is a generalized linear classifier that performs binary classification on data in a supervised learning manner. Its decision-making boundary is the maximum-margin hyperplane for solving learning samples. SVM is a sparse and robust classifier that uses the hinge loss function to calculate empirical risks and adds regularization items to the problem-solving system to relieve structural risks. The LinearSVC algorithm of Spark introduces two optimization policies: reducing the times of invoking the f functions (distributed computing of loss and gradient of the target functions) through algorithm principle optimization, and accelerating convergence by increasing momentum parameter updates.

• Decision Tree

  The Decision Tree algorithm is widely used in fields such as machine learning and computer vision for classification and regression. It trains a binary tree to obtain a classification model or regression model based on given sample data that contains feature vectors and label values. The output model can predict the label value with the highest probability after the feature vectors are input.

• Linear Regression

  Regression algorithms are supervised learning algorithms used to find possible relationships between the independent variable X and the observable variable Y. If the observable variable is continuous, it is called "regression". In machine learning, Linear Regression uses a linear model to model the relationship between the independent variable X and the observable variable Y. The unknown model parameters are estimated from the training data.

• Logistic Regression

  Logistic Regression is a classification method that uses a linear model to model the relationship between the independent variable X and the observable variable Y. The unknown model parameters are estimated from the training data.

• XGBoost

  XGBoost is a deeply-optimized distributed gradient boosting algorithm library that is efficient, flexible, and portable. It implements machine learning algorithms in the framework of gradient boosting, and provides a parallel tree boosting algorithm, which can quickly and accurately solve many data science problems.

• KNN

  K-nearest neighbors (KNN) is a non-parametric algorithm in machine learning that is used to find k samples closest to a given sample. It can be used for classification, regression, and information retrieval.

Programming Example

This example describes the programming with the GBDT algorithm.

GBDT has two types of model interfaces: ML Classification API and ML Regression API.

Figure 1 shows the time sequence of the GBDT classification model.

Figure 1 Time sequence of the GBDT classification model

ML Classification API

ML Regression API

• Function description

  Output the GBDT regression model after you input sample data in dataset format and call the training API.

• Input and output
  1. Package name: package org.apache.spark.ml.classification
  2. Class name: GBTRegressor
  3. Method name: fit
  4. Input: training sample data (Dataset[_]). The following are mandatory fields.

     Parameter	Value Type	Default Value	Description
     labelCol	Double	label	Predicted label
     featuresCol	Vector	features	Feature label

  5. Input: the model parameters of the fit API paramMap, paramMaps, firstParamPair, otherParamPairs, which are described as follows:

     Parameter	Value Type	Example	Description
     paramMap	ParamMap	ParamMap(A.c -> b)	Assigns the value of b to the parameter c of model A.
     paramMaps	Array[ParamMap]	Array[ParamMap](n)	Generates n parameter lists for the ParamMap model.
     firstParamPair	ParamPair	ParamPair(A.c, b)	Assigns the value of b to the parameter c of model A.
     otherParamPairs	ParamPair	ParamPair(A.e, f)	Assigns the value of f to the parameter e of model A.

  6. Parameters optimized based on native algorithms

     def setCheckpointInterval(value: Int): GBTRegressor.this.type
     def setFeatureSubsetStrategy(value: String): GBTRegressor.this.type
     def setFeaturesCol(value: String): GBTRegressor
     def setImpurity(value: String): GBTRegressor.this.type
     def setLabelCol(value: String): GBTRegressor
     def setLossType(value: String): GBTRegressor.this.type
     def setMaxBins(value: Int): GBTRegressor.this.type
     def setMaxDepth(value: Int): GBTRegressor.this.type
     def setMaxIter(value: Int): GBTRegressor.this.type
     def setMinInfoGain(value: Double): GBTRegressor.this.type
     def setMinInstancesPerNode(value: Int): GBTRegressor.this.type
     def setPredictionCol(value: String): GBTRegressor
     def setSeed(value: Long): GBTRegressor.this.type
     def setStepSize(value: Double): GBTRegressor.this.type
     def setSubsamplingRate(value: Double): GBTRegressor.this.type

     

     An example is provided as follows:

     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

     import org.apache.spark.ml.param.{ParamMap, ParamPair} val gbdt = new GBTRegressor() // Define the regression model. // Define the def fit(dataset: Dataset[_], paramMap: ParamMap) API parameter. val paramMap = ParamMap(gbdt.maxDepth -> maxDepth) .put(gbdt.maxIter, maxIter) // Define the def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): API parameter. val paramMaps: Array[ParamMap] = new Array[ParamMap](2) for (i <- 0 to 2) { paramMaps(i) = ParamMap(gbdt.maxDepth -> maxDepth) .put(gbdt.maxIter, maxIter) }//Assign a value to paramMaps. // Define the def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*) API parameter. val maxDepthParamPair = ParamPair(gbdt.maxDepth, maxDepth) val maxIterParamPair = ParamPair(gbdt.maxIter, maxIter) val maxBinsParamPair = ParamPair(gbdt.maxBins, maxBins) // Call the fit APIs. model = gbdt.fit(trainingData) // Return GBTRegressionModel. model = gbdt.fit(trainingData, paramMap) // Return GBTRegressionModel. models = gbdt.fit(trainingData, paramMaps) // Return Seq[GBTRegressionModel]. model = gbdt.fit(trainingData, maxDepthParamPair, maxIterParamPair, maxBinsParamPair) //Return GBTRegressionModel.

  7. Output: GBDT regression model (GBTRegressionModel or Seq[GBTRegressionModel]). The following table lists the field output in model prediction.

     Parameter	Value Type	Default Value	Description
     predictionCol	Double	prediction	Predicted label

     Figure 2 shows the time sequence of the GBDT regression model.

     Figure 2 Time sequence of the GBDT regression model
     
• Example
  fit(dataset: Dataset[_]): GBTRegressionModel example:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

  import org.apache.spark.ml.Pipeline import org.apache.spark.ml.evaluation.RegressionEvaluator import org.apache.spark.ml.feature.VectorIndexer import org.apache.spark.ml.regression.{GBTRegressionModel, GBTRegressor} // Load and parse the data file, converting it to a DataFrame. val data = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt") // Automatically identify categorical features, and index them. // Set maxCategories so features with > 4 distinct values are treated as continuous. val featureIndexer = new VectorIndexer() .setInputCol("features") .setOutputCol("indexedFeatures") .setMaxCategories(4) .fit(data) // Split the data into training and test sets (30% held out for testing). val Array(trainingData, testData) = data.randomSplit(Array(0.7, 0.3)) // Train a GBT model. val gbt = new GBTRegressor() .setLabelCol("label") .setFeaturesCol("indexedFeatures") .setMaxIter(10) // Chain indexer and GBT in a Pipeline. val pipeline = new Pipeline() .setStages(Array(featureIndexer, gbt)) // Train model. This also runs the indexer. val model = pipeline.fit(trainingData) // Make predictions. val predictions = model.transform(testData) // Select example rows to display. predictions.select("prediction", "label", "features").show(5) // Select (prediction, true label) and compute test error. val evaluator = new RegressionEvaluator() .setLabelCol("label") .setPredictionCol("prediction") .setMetricName("rmse") val rmse = evaluator.evaluate(predictions) println("Root Mean Squared Error (RMSE) on test data = " + rmse) val gbtModel = model.stages(1).asInstanceOf[GBTRegressionModel] println("Learned regression GBT model:\n" + gbtModel.toDebugString)

• Result

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

  Root Mean Squared Error (RMSE) on test data = 0.0 Learned regression GBT model: GBTRegressionModel (uid=gbtr_842c8acff963) with 10 trees Tree 0 (weight 1.0): If (feature 434 <= 70.5) If (feature 99 in {0.0,3.0}) Predict: 0.0 Else (feature 99 not in {0.0,3.0}) Predict: 1.0 Else (feature 434 > 70.5) Predict: 1.0 Tree 1 (weight 0.1): Predict: 0.0 Tree 2 (weight 0.1): Predict: 0.0 Tree 3 (weight 0.1): Predict: 0.0 Tree 4 (weight 0.1): Predict: 0.0 Tree 5 (weight 0.1): Predict: 0.0 Tree 6 (weight 0.1): Predict: 0.0 Tree 7 (weight 0.1): Predict: 0.0 Tree 8 (weight 0.1): Predict: 0.0 Tree 9 (weight 0.1): Predict: 0.0