pig-on-tez or hive-on-tez 报错
2015-09-16 15:28
435 查看
当pig-0.15或者hive-1.1-cdh5.4.0使用tez运行是 会报如下的错误,发现申请内存为-1,所以报错
发现无论是pig还是hive的相关默认配置如下
pig的
mapreduce.map.memory.mb=-1
mapreduce.reduce.memory.mb=-1
hive的
mapreduce.map.memory.mb=-1
mapreduce.reduce.memory.mb=-1
hive.tez.container.size=-1
故使用tez的时候
pig 设置
mapreduce.map.memory.mb 2048
mapreduce.reduce.memory.mb 2048
hive设置
hive.tez.container.size 2048
发现无论是pig还是hive的相关默认配置如下
pig的
mapreduce.map.memory.mb=-1
mapreduce.reduce.memory.mb=-1
hive的
mapreduce.map.memory.mb=-1
mapreduce.reduce.memory.mb=-1
hive.tez.container.size=-1
故使用tez的时候
pig 设置
mapreduce.map.memory.mb 2048
mapreduce.reduce.memory.mb 2048
hive设置
hive.tez.container.size 2048
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=-1, maxMemory=30720 at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:203) at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:96) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:501) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.allocate(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=-1, maxMemory=30720 at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:203)
相关文章推荐
- 分享Hive的一份胶片资料
- Hadoop生态上几个技术的关系与区别:hive、pig、hbase 关系与区别
- IE7降低内存和降低CPU的几个技巧
- 如何高效的使用内存
- DOS下内存的配置
- XP/win2003下发现1G的内存比512M还慢的解决方法
- PowerShell实现动态获取当前脚本运行时消耗的内存
- C#实现把dgv里的数据完整的复制到一张内存表的方法
- SQL语句实现查询SQL Server内存使用状况
- C语言内存对齐实例详解
- 深入学习C语言中memset()函数的用法
- 全局变量与局部变量在内存中的区别详细解析
- VB读取线程、句柄及写入内存的API代码实例
- php运行提示:Fatal error Allowed memory size内存不足的解决方法
- IE浏览器IFrame对象内存不释放问题解决方法
- C#之CLR内存深入分析
- JavaScript 变量、作用域及内存
- JavaScript避免内存泄露及内存管理技巧
- c++实现逐行读取配置文件写入内存的示例