K-means

K-means为ML API。

• 输入输出
  1. 包名：package org.apache.spark.ml.clustering
  2. 类名：KMeans
  3. 方法名：fit
  4. 输入：Dataset[_]，训练样本数据，必须字段如下

     参数名称	取值类型	缺省值	描述
     featuresCol	Vector	features	特征标签

  5. 基于原生算法优化的参数

     def setFeaturesCol(value: String): KMeans.this.type
     def setPredictionCol(value: String): KMeans.this.type
     def setK(value: Int): KMeans.this.type
     def setInitMode(value: String): KMeans.this.type
     def setInitSteps(value: Int): KMeans.this.type
     def setMaxIter(value: Int): KMeans.this.type
     def setThreshold(value: Double): KMeans.this.type
     def setTol(value: Double): KMeans.this.type
     def setSeed(value: Long): KMeans.this.type

  6. 新增算法参数

     参数名称	参数含义	取值类型
     sampleRate	每一轮迭代使用的数据占全量数据集的比例	0~1[Double]
     optMethod	样本数据采样触发开关	default/allData[String]

     

     参数及fit代码接口示例：

     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

     import org.apache.spark.ml.param.{ParamMap, ParamPair} val kmeans = new MlKMeans() //定义def fit(dataset: Dataset[_], paramMap: ParamMap) 接口参数 val paramMap = ParamMap(kmeans.initSteps -> initSteps) .put(k-means.maxIter, maxIter) // 定义def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): 接口参数 val paramMaps: Array[ParamMap] = new Array[ParamMap](2) for (i <- 0 to 2) { paramMaps(i) = ParamMap(kmeans.initSteps -> initSteps) .put(kmeans.maxIter, maxIter) }//对paramMaps进行赋值 // 定义def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*) 接口参数 val initStepsParamPair = ParamPair(kmeans.initSteps, initSteps) val maxIterParamPair = ParamPair(kmeans.maxIter, maxIter) val tolParamPair = ParamPair(kmeans.tol, tol) // 调用各个fit接口 model = kmeans.fit(trainingData) model = kmeans.fit(trainingData, paramMap) models = kmeans.fit(trainingData, paramMaps) model = kmeans.fit(trainingData, initStepsParamPair, maxIterParamPair, tolParamPair)

  7. 输出：KMeansModel，KMeans聚类模型，模型预测时的输出字段如下

     参数名称	取值类型	缺省值	描述
     predictionCol	Int	prediction	predictionCol

• 使用样例

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

  import org.apache.spark.ml.clustering.KMeans import org.apache.spark.ml.evaluation.ClusteringEvaluator // Loads data. val dataset = spark.read.format("libsvm").load("data/mllib/sample_kmeans_data.txt") // Trains a k-means model. val kmeans = new KMeans().setK(2).setSeed(1L) val model = kmeans.fit(dataset) // Make predictions val predictions = model.transform(dataset) // Evaluate clustering by computing Silhouette score val evaluator = new ClusteringEvaluator() val silhouette = evaluator.evaluate(predictions) println(s"Silhouette with squared euclidean distance = $silhouette") // Shows the result. println("Cluster Centers: ") model.clusterCenters.foreach(println)