大数据hadoop学习之---Yarn 体系架…

Apache Hadoop NextGen MapReduce (YARN)

MapReduce has undergone a complete overhaul in hadoop-0.23 and we
now have, what we call, MapReduce 2.0 (MRv2) or YARN.

The fundamental idea of MRv2 is to split up the two major
functionalities of the JobTracker, resource management and job
scheduling/monitoring, into separate daemons. The idea is to have a
global ResourceManager (RM) and per-application
ApplicationMaster (AM). An application is either a single
job in the classical sense of Map-Reduce jobs or a DAG of jobs.

The ResourceManager and per-node slave, the NodeManager
(NM), form the data-computation framework. The
ResourceManager is the ultimate authority that arbitrates resources
among all the applications in the system.

The per-application ApplicationMaster is, in effect, a framework
specific library and is tasked with negotiating resources from the
ResourceManager and working with the NodeManager(s) to execute and
monitor the tasks.



The ResourceManager has two main components: Scheduler and
ApplicationsManager.

The Scheduler is responsible for allocating resources to the
various running applications subject to familiar constraints of
capacities, queues etc. The Scheduler is pure scheduler in the
sense that it performs no monitoring or tracking of status for the
application. Also, it offers no guarantees about restarting failed
tasks either due to application failure or hardware failures. The
Scheduler performs its scheduling function based the resource
requirements of the applications; it does so based on the abstract
notion of a
resource Container which
incorporates elements such as memory, cpu, disk, network etc. In
the first version,
only memory is
supported.

The Scheduler has a pluggable policy plug-in, which is responsible
for partitioning the cluster resources among the various queues,
applications etc. The current Map-Reduce schedulers such as the
CapacityScheduler and the FairScheduler would be some examples of
the plug-in.

The CapacityScheduler supports hierarchical
queues to allow for more predictable sharing
of cluster resources

The ApplicationsManager is responsible for accepting
job-submissions, negotiating the first container for executing the
application specific ApplicationMaster and provides the service for
restarting the ApplicationMaster container on failure.

The NodeManager is the per-machine framework agent who is
responsible for containers, monitoring their resource usage (cpu,
memory, disk, network) and reporting the same to the
ResourceManager/Scheduler.

The per-application ApplicationMaster has the responsibility of
negotiating appropriate resource containers from the Scheduler,
tracking their status and monitoring for progress.

MRV2 maintains API
compatibility with previous stable release
(hadoop-1.x). This means that all Map-Reduce jobs should still run
unchanged on top of MRv2 with just a recompile.

个人翻译和理解:

MapReduce
到0.23版本已经终结了,我们现在有了2.0版本的mapreduce,也就是Yarn.

yarn 的基本原理就是分割JobTracker的两大功能:
资源管理与任务调度及监控,将这两个功能分割为

单独的守护进程.

基本的思想就是有一个全局的资源管理器和
基于应用的appMaster(每个应用一个appmaster),

这里提及的应用就是传统意义上的map-reduce任务或者DAG任务

资源管理器节点ResourceManger以及每个slave节点即NodeManager
组成数据计算框架.

ResourceManager管理器节点对资源的丛书有完全的分配权,决定资源归属于系统中的哪个应用.

基于应用的applicationMaster
实际上就是一个特定的框架工具包,负责和ResourceManager协商获取

资源,将ResourceManager分配给的NodeManager节点的资源(slot)进行二次分配.

将任务分配给获取到的节点资源,将任务分发给这些节点,监控他们的任务执行情况.

ResourceManager
有两个主要的组件:
调度器和applicationManager

调度器负责将资源分配给多个主题相似的应用(主题相似指的是队列约束容量等等).

调度器就是一个纯粹的调度器,也就是说不负责监控和追踪应用的执行状态.

同时,不负责重启因为硬件或者应用本身出错导致的执行失败的应用.

调度器基于应用本身资源请求实现调度功能.

同样的调度器基于应用本身的资源请求

抽象出来资源容器的概念,资源容器包含了内存,cpu,磁盘网络等等元素,但是

第一个版本仅仅只支持内存.

调度器支持插件式的嵌入策略,资源调度器插件算法负责多个应用,多个队列之间,将

集群资源进行分区.

目前而言,mapreduce
调度器比如
容量调度器和公平调度器是可插件式插入的例子.

资源调度器支持分级队列,对集群资源做更加可预见的分配.

applicationManager
负责接收任务的提交,协商第一个容器,

为执行任务的应用指定ApplicationMaster,同时当applicationMaster容器失败后重启

NodeManager
是资源容器的代理框架,安装在每个从节点中,负责监控当前机器的

资源使用率(cpu,内存,磁盘,网络)
,将节点中的资源使用率报告给ResourceManager
的资源调度器

基于应用的applicationMaster 负责从
资源调度器的资源容器协商获取资源,追踪状态即监控进度等.

MRV2 负责维护 和hadoop1
的版本兼容性,也就是说,所有用hadoop1
api写的map-reduce任务,仅仅需要重新编译就可以允许在

MRV2上.

Yarn
就是为了分离JobTracker的两大职责:
资源管理及任务调度.

其中资源管理引进了ResourceManager,统一管理资源,谁需要资源,需要通过ResourceManager申请

任务的调度有通用的接口,支持Map-Reduce,DAG,Spark,Storm等,只要符合规范,实现指定的接口按照,Yarn既定流程进行实现,就能按照

Yarn的设计初衷来走.这样就统一了企业资源分配和任务调度平台,大大提高集群的利用率.



转发至微博
 



转发至微博

阅读(0)|
评论(0)

|
       



用微信  “扫一扫”

将文章分享到朋友圈。

 



用易信  “扫一扫”

将文章分享到朋友圈。

 

喜欢 推荐 0人  | 
转载

 

历史上的今天

最近读者

热度

在LOFTER的更多文章

关闭

玩LOFTER，免费冲印20张照片，人人有奖！     我要抢>

评论

this.p={ m:2,
b:2,
loftPermalink:'',
id:'fks_087070082094081074082087085075072084080068086081080065084095',
blogTitle:'大数据hadoop学 之---Yarn 体系架构英文原版及个人翻译和理解',
blogAbstract:'

Apache Hadoop NextGen MapReduce (YARN)

MapReduce has undergone a complete overhaul in hadoop-0.23 and we
now have, what we call, MapReduce 2.0 (MRv2) or YARN.

The
fundamental idea of MRv2 is to split up the two major
functionalities of the JobTracker, resource management and job
scheduling/monitoring, into separate daemons. The idea is to have a
global ResourceManager (RM) and per-application
ApplicationMaster (

', blogTag:'', blogUrl:'blog/static/2172762092015124244789',
isPublished:1, istop:false, type:0, modifyTime:1424758045743,
publishTime:1424758045701,
permalink:'blog/static/2172762092015124244789', commentCount:0,
mainCommentCount:0, recommendCount:0, bsrk:-100, publisherId:0,
recomBlogHome:false, currentRecomBlog:false, attachmentsFileIds:[],
vote:{}, groupInfo:{}, friendstatus:'none',
followstatus:'unFollow', pubSucc:'', visitorProvince:'',
visitorCity:'', visitorNewUser:false, postAddInfo:{}, mset:'000',
mcon:'', srk:-100, remindgoodnightblog:false, isBlackVisitor:false,
isShowYodaoAd:false,
hostIntro:'JAVA软件工程师,有扎实的Java基础,熟悉JavaEE技术,对框架的底层原理熟悉,学习能力强。',
hmcon:'0', selfRecomBlogCount:'0', lofter_single:'' }

{list a as x}
{if !!x}
{if
x.visitorName==visitor.userName}

{else}


{/if}
{if x.moveFrom=='wap'}   {elseif
x.moveFrom=='iphone'}   {elseif
x.moveFrom=='android'}   {elseif
x.moveFrom=='mobile'}   {/if}
${fn(x.visitorNickname,8)|escape}

{/if} {/list}

{if !!a}
${fn(a.nickname,8)|escape}
${a.selfIntro|escape}{if
great260}${suplement}{/if}

 

{/if}