第30课 Master的注册机制和状态管理解密
2016-07-02 15:35
429 查看
一 、master对其他组件注册的处理
1, master接受注册的对象主要就是:driver,application,worker;需要补充说明executor不会注册给master,executor是注册给driver中的schedulerbackbend的;
2, worker是再启动后主动向master注册的,所以如果在生产环境下加入新的worker到已经正在运行的Spark集群上,此时不需要重新启动spark集群就能够使用新加入的worker以提升处理能力;
3, master在接收到worker注册的请求后,首先会判断一下当前的master是否是standby的模式,如果是的话就不处理;然后会判断当前master的内存数据结构idToWorker中是否已经有该worker的注册信息,如果有的话此时不会重复注册;
4, master如果决定接收注册的worker,首先会创建workerInfo对象来保存注册的worker信息;
然后调用registerWorker来执行具体的注册过程,如果worker的状态是否是dead的状态则直接过滤掉,对于unknown装的内容调用removeWorker进行清理(包括清理worker下的executors和driver)
5, 注册时候是先注册driver然后在注册application;
二 master对driver和executor状态变化的处理
1, 对driver状态变化的处理
2, Executor挂掉的时候系统会尝试一定次数的重启(最多重试10次重启)
1, master接受注册的对象主要就是:driver,application,worker;需要补充说明executor不会注册给master,executor是注册给driver中的schedulerbackbend的;
2, worker是再启动后主动向master注册的,所以如果在生产环境下加入新的worker到已经正在运行的Spark集群上,此时不需要重新启动spark集群就能够使用新加入的worker以提升处理能力;
<code class="hljs perl has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">case RegisterWorker( id, workerHost, workerPort, workerRef, cores, memory, workerUiPort, publicAddress)=> { logInfo(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Registeringworker <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">%s</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">%d</span> with <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">%d</span> cores, <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">%s</span> RAM"</span>.<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">format</span>( workerHost, workerPort, cores, Utils.megabytesToString(memory))) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">state</span> == RecoveryState.STANDBY) { context.reply(MasterInStandby) } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (idToWorker.contains(id)) { context.reply(RegisterWorkerFailed(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Duplicate worker ID"</span>)) } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> { val worker = new WorkerInfo(id, workerHost, workerPort, cores, memory, workerRef, workerUiPort, publicAddress) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (registerWorker(worker)) { persistenceEngine.addWorker(worker) context.reply(RegisteredWorker(self, masterWebUiUrl)) schedule() } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> { val workerAddress = worker.endpoint.address logWarning(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Workerregistration failed. Attempted to re-register worker at same "</span> + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"address:"</span> + workerAddress) context.reply(RegisterWorkerFailed(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Attempted to re-register worker at sameaddress: "</span> + workerAddress)) } }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li></ul>
3, master在接收到worker注册的请求后,首先会判断一下当前的master是否是standby的模式,如果是的话就不处理;然后会判断当前master的内存数据结构idToWorker中是否已经有该worker的注册信息,如果有的话此时不会重复注册;
4, master如果决定接收注册的worker,首先会创建workerInfo对象来保存注册的worker信息;
<code class="hljs fsharp has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span>[spark] <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> WorkerInfo( <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> id: String, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> host: String, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> port: Int, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> cores: Int, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> memory: Int, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-bo cc38 x;">val</span> endpoint: RpcEndpointRef, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> webUiPort: Int, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> publicAddress: String) extends Serializable { }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li></ul>
然后调用registerWorker来执行具体的注册过程,如果worker的状态是否是dead的状态则直接过滤掉,对于unknown装的内容调用removeWorker进行清理(包括清理worker下的executors和driver)
5, 注册时候是先注册driver然后在注册application;
二 master对driver和executor状态变化的处理
1, 对driver状态变化的处理
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">case DriverState<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ERROR</span> | DriverState<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FINISHED</span> | DriverState<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.KILLED</span> | DriverState<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FAILED</span> => removeDriver(driverId, state, exception)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
2, Executor挂掉的时候系统会尝试一定次数的重启(最多重试10次重启)
相关文章推荐
- 用html写一个简短的员工注册界面
- 别因为要学的太多反而压垮自己
- 堆和栈
- android sudio 如何获取sha1与md5值
- 关于C语言中的变量
- Windows 环境下的Oracle 编程
- 几种方法
- unity3d优化-代码篇(不定期更新)
- ubuntu jenkins 构建报“Failure to find xxx.jar apache.org/maven2 was cached in the local repository"错误
- 关于Debug和Release之本质区别的讨论
- 如何学好MFC
- Android屏幕知识点
- SSI
- 关于动静态链接库的使用,dll调用,等
- rabbitmq——用户管理
- Canvas改变原点绕中心旋转
- Supervisor-进程监控自动重启
- 关于回调的一些东东
- Poj 1068:Parencodings
- setNeedsLayout和layoutIfNeeded的区别