Spark闭包与序列化

本文原文出处: http://blog.csdn.net/bluishglc/article/details/50945032 严禁任何形式的转载，否则将委托CSDN官方维护权益！

Spark的官方文档再三强调那些将要作用到RDD上的操作，不管它们是一个函数还是一段代码片段，它们都是“闭包”，Spark会把这个闭包分发到各个worker节点上去执行，这里涉及到了一个容易被忽视的问题：闭包的“序列化”。

显然，闭包是有状态的，这主要是指它牵涉到的那些自由变量以及自由变量依赖到的其他变量，所以，在将一个简单的函数或者一段代码片段（就是闭包）传递给类似RDD.map这样的操作前，Spark需要检索闭包内所有的涉及到的变量（包括传递依赖的变量），正确地把这些变量序列化之后才能传递到worker节点并反序列化去执行。如果在涉及到的所有的变量中有任何不支持序列化或没有指明如何序列化自己时，你就会遇到这样的错误：

org.apache.spark.SparkException: Task not serializable

在下面的例子中，我们从kafka中持续地接收json消息，并在spark-streaming中将字符串解析成对应的实体：

<code class="language-scala hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">object</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">App</span> {</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> config = ConfigFactory.load(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"my-streaming.conf"</span>)
<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Person</span> <span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(firstName: String,lastName: String)</span></span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> main(args: Array[String]) {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> zkQuorum = config.getString(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka.zkQuorum"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> myTopic = config.getString(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka.myTopic"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> myGroup = config.getString(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka.myGroup"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> conf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"my-streaming"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> ssc = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> StreamingContext(conf, Seconds(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> lines = KafkaUtils.createStream(ssc, zkQuorum, myGroup, Map(myTopic -> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//this val is a part of closure, and it's not serializable!</span>
implicit <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> formats = DefaultFormats
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> parser(json: String) = parse(json).extract[Person].firstName
lines.map(_._2).map(parser).print
....
ssc.start()
ssc.awaitTerminationOrTimeout(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10000</span>)
ssc.stop()
}

}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li></ul>

这段代码在执行时就会报如下错误：

org.apache.spark.SparkException: Task not serializableCaused by: java.io.NotSerializableException: org.json4s.DefaultFormats$

问题的症结就在于：闭包没有办法序列化。在这个例子里，闭包的范围是：函数parser以及它所依赖的一个隐式参数: formats , 而问题就出在这个隐式参数上， 它的类型是DefaultFormats，这个类没有提供序列化和反序列自身的说明，所以Spark无法序列化formats，进而无法将task推送到远端执行。

隐式参数formats是为extract准备的，它的参数列表如下：

<code class="hljs markdown has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">org.json4s.ExtractableJsonAstNode#extract[<span class="hljs-link_label" style="box-sizing: border-box;">A</span>](<span class="hljs-link_url" style="box-sizing: border-box;">implicit formats: Formats, mf: scala.reflect.Manifest[A]</span>): A = ...</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

找到问题的根源之后就好解决了。实际上我们根本不需要序列化formats， 对我们来说，它是无状态的。所以，我们只需要把它声明为一个全局静态的变量就可以绕过序列化。所以改动的方法就是简单地把

implicit val formats = DefaultFormats

的声明从方法内部迁移到App
Object的字段位置上即可。

<code class="language-scala hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">object</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">App</span> {</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> config = ConfigFactory.load(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"my-streaming.conf"</span>)
<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">case</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">Person</span> <span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(firstName: String,lastName: String)</span></span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//As Object field, global, static, no need to serialize</span>
implicit <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> formats = DefaultFormats

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> main(args: Array[String]) {
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> zkQuorum = config.getString(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka.zkQuorum"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> myTopic = config.getString(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka.myTopic"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> myGroup = config.getString(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka.myGroup"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> conf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"my-streaming"</span>)
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> ssc = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> StreamingContext(conf, Seconds(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> lines = KafkaUtils.createStream(ssc, zkQuorum, myGroup, Map(myTopic -> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> parser(json: String) = parse(json).extract[Person].firstName
lines..map(_._2).map(parser).print
....
ssc.start()
ssc.awaitTerminationOrTimeout(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10000</span>)
ssc.stop()
}

}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul>

这里再提供另外一个很好的例子：



这个例子很好演示了解决类似问题的方案：“把类成员变量拷贝一份到闭包中” ，不然整个对象都需要被序列化!

最后我们来总结一下应该如何正确的处理Spark Task闭包的序列化问题。首先你需要对Task涉及的闭包的边界要有一个清晰的认识，要尽量地控制闭包的范围和牵涉到的自由变量，一个非常值得警惕的地方是：尽量不要在闭包中直接引用一个类的成员变量和函数，这样会导致整个类实例被序列化。这样的例子在Spark文档中也有提及，如下：

<code class="language-scala hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyClass</span> {</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> func1(s: String): String = { ... }
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> doStuff(rdd: RDD[String]): RDD[String] = { rdd.map(func1) }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

然后，一个好的组织代码的方式是：除了那些很短小的函数，尽量把复杂的操作封装到全局单一的函数体：全局静态方法或者函数对象

如果确实需要某个类的实例参与到计算过程中，则要作好相关的序列化工作。