Spark Scala 分组排序取TopN

1.输入

c1 85
c2 77
c3 88
c1 22
c1 66
c3 95
c3 54
c2 91
c2 66
c1 54
c1 65
c2 41
c4 65

2.代码实现

import org.apache.spark.{SparkContext, SparkConf}

/**
* User:leen
* Date:2017/10/18 0018
* Time:19:47
* 分组之后取出每组的TopN
*/
object GroupTopN1 {

def main(args: Array[String]) {
val conf = new SparkConf().setAppName("GroupTopN1").setMaster("local")
val sc = new SparkContext(conf)
val lines = sc.textFile("C:\\Users\\leen\\Desktop\\group.txt")
//拆分为Tuple2
val classScores = lines.map(line => (line.split(" ")(0), line.split(" ")(1).toInt))
//分组
val group = classScores.groupByKey()
//针对分组对value排序,返回Tuple2
val groupSort = group.map(css => {
val c = css._1
val ss = css._2
val sortScore = ss.toList.sortWith(_ > _).take(3) //降序，取Top3
(c, sortScore)
})
//遍历输出
groupSort.foreach(v => {
print(v._1 + " : ")
v._2.foreach(ss => print(ss + "\t"))
println()
})

sc.stop()
}
}

3.输出结果

c2 : 91 77  66
c3 : 95 88  54
c4 : 65
c1 : 85 66  65

4.JAVA版本参考地址：