Hadoop & HBase 故障排除
2016-07-11 11:05
260 查看
Hadoop & HBase 故障排除
接到报警反馈HBase故障。1.上线检查HBase日志信息,有如下报错信息,无法链接RegionServer
2015-06-23 08:50:39,219 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInM
eta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=Colander-hb03, port=60020}, attempt=81 of 100 fa
iled; retrying after sleep of 32044 because: Call to Colander-hb03/10.161.181.208:60020 failed on local exception: java.io
.IOException: Connection reset by peer
2.决定重启HBase,执行stop-hbase.sh很长时间无法关闭,使用kill杀掉相关进程,再start-hbase.sh开启HBase。
3.开启后无法正常工作,日志未出现任何错误,HMaster显示available,但是RegionServer显示master not available
HMaster日志
2015-06-23 09:41:43,644 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: A master is now available
2015-06-23 09:41:43,651 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Master=Colander-hb01,60000,1435023696774
RegionServer日志
2015-06-23 09:55:08,156 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Master isn’t available yet, retrying
2015-06-23 09:55:08,357 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at Colander-hb01,60000,1435024143478
4.HMaster 和 RegionServer 都运行正常,问题调查无法进展下去。换个思路,看是否HDFS故障。
5.检查Hadoop日志,发现确实有问题
2015-06-23 09:40:36,292 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.168.234.177:50010, storageID=DS-1368639422-10.168.234.177-50010-1417075244953, infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:295)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
at java.lang.Thread.run(Thread.java:745)
6.重启Hadoop DFS
./bin/stop-dfs.sh; ./bin/start-dfs.sh
7.HBase依然运行异常,重启后恢复正常。
相关文章推荐
- 性能测试中用LambdaProbe监控Tomcat
- .NET跨平台之旅:在生产环境中上线第一个运行于Linux上的ASP.NET Core站点
- 关于多线程
- Opencv图像识别从零到精通(2)-----准备知识
- libcurl CURLOPT_WRITEFUNCTION注意事项
- AutoScrollTopBottomView
- Linux内存空间的分配
- 双向认证第一步:Mac OS X中Apache开启ssl
- eclipse远程调试Tomcat方法(测试成功并且说说遇到的坑)
- Linux内存管理之页面回收【转】
- hadoop: hdfs API示例
- linux内核分析之进程地址空间【转】
- Hadoop&HBase部署文档
- 嵌入式LINUX调试器GDB
- IBM BigInsights 4.1 发布了 ,对Hadoop的支持更加深入!
- 【Linux】shell变量篇(收藏)
- hadoop源码分析解读入门
- Tomcat 指定的服务已存在 Failed installing 'Tomcat8'
- .properties 文件
- Shell脚本8种字符串截取方法总结