使用flume中遇见的问题整理
2013-02-19 15:48
281 查看
先发一下牢骚: 这玩意真恶心遇见个问题都搜不到。。。。 要么就是英文的,比啃牛皮还难受。 所以写一个汇集, 要是大家有新问题或者别的解决方法,有什么不对的地方,尽情留言拍砖,大家共同学习进步
1 开发的jar包 是取得 flume安装目录下的lib内的所有
2 提示log4j WARN
解决方法:在src下补充log4j.properties 文件
3 运行程序向avor的source里面写东西,提示异常 NettyAvroRpcClient { host: n1, port: 44444 }: RPC connection error
完整异常:
Exception in thread "main" org.apache.flume.FlumeException: NettyAvroRpcClient { host: n1, port: 44444 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:128)
at cn.vko.flume.MyRpcClientFacade.init(MyRpcClientFacade.java:22)
at cn.vko.flume.ClientApp.main(ClientApp.java:10)
Caused by: java.io.IOException: Error connecting to /192.168.1.16:44444
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 6 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
解决方法: 这是flume-ng 没有启动,检查是否启动着java进程 Application 。。。
4 运行程序时,$FLUME_HOME/logs/flume.log 中异常Flume wasn't able to parse timestamp header in the event to resolve time based bucketing:
19 Feb 2013 14:18:59,799 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp
header (for example using TimestampInterceptor source interceptor).
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using
TimestampInterceptor source interceptor).
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:160)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:343)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:397)
... 3 more
Caused by: java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:375)
at java.lang.Long.valueOf(Long.java:525)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:158)
... 5 more
解决方法: 这是HDFS sink 提示的异常,这是因为在hdfs路径中有时间参数,然后取event中的header的timestamp 参数,取不到提示的异常;所以解决方法是
1 检查配置文件conf/flume-conf.properties 中是否用到了时间参数,去掉他们。
2 或者在提交event的时候添加header的timestamp 参数,如下:
Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
Map<String, String> headers = new HashMap<String, String>();
headers.put("timestamp", String.valueOf(System.currentTimeMillis()));/**/
event.setHeaders(headers);
// Send the event
try {
client.append(event);
} catch (EventDeliveryException e) {
// clean up and recreate the client
client.close();
client = null;
client = RpcClientFactory.getDefaultInstance(hostname, port);
}
3 使用多路符合那种方案部署时怎么弄,不是怎么清楚呢。。。 待查清
(未完待完善。。。。。。 欢迎补充。。。 )
1 开发的jar包 是取得 flume安装目录下的lib内的所有
2 提示log4j WARN
解决方法:在src下补充log4j.properties 文件
3 运行程序向avor的source里面写东西,提示异常 NettyAvroRpcClient { host: n1, port: 44444 }: RPC connection error
完整异常:
Exception in thread "main" org.apache.flume.FlumeException: NettyAvroRpcClient { host: n1, port: 44444 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:128)
at cn.vko.flume.MyRpcClientFacade.init(MyRpcClientFacade.java:22)
at cn.vko.flume.ClientApp.main(ClientApp.java:10)
Caused by: java.io.IOException: Error connecting to /192.168.1.16:44444
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 6 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
解决方法: 这是flume-ng 没有启动,检查是否启动着java进程 Application 。。。
4 运行程序时,$FLUME_HOME/logs/flume.log 中异常Flume wasn't able to parse timestamp header in the event to resolve time based bucketing:
19 Feb 2013 14:18:59,799 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp
header (for example using TimestampInterceptor source interceptor).
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using
TimestampInterceptor source interceptor).
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:160)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:343)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:397)
... 3 more
Caused by: java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:375)
at java.lang.Long.valueOf(Long.java:525)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:158)
... 5 more
解决方法: 这是HDFS sink 提示的异常,这是因为在hdfs路径中有时间参数,然后取event中的header的timestamp 参数,取不到提示的异常;所以解决方法是
1 检查配置文件conf/flume-conf.properties 中是否用到了时间参数,去掉他们。
2 或者在提交event的时候添加header的timestamp 参数,如下:
Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
Map<String, String> headers = new HashMap<String, String>();
headers.put("timestamp", String.valueOf(System.currentTimeMillis()));/**/
event.setHeaders(headers);
// Send the event
try {
client.append(event);
} catch (EventDeliveryException e) {
// clean up and recreate the client
client.close();
client = null;
client = RpcClientFactory.getDefaultInstance(hostname, port);
}
3 使用多路符合那种方案部署时怎么弄,不是怎么清楚呢。。。 待查清
(未完待完善。。。。。。 欢迎补充。。。 )
相关文章推荐
- [Intellij] Intellij IDEA 使用中遇见的问题
- dll中使用ado时遇见的问题
- 使用angularJS遇见的一些问题的解决方案
- 使用uri api调起本地百度地图遇见的问题
- NGUI使用过程中遇见的问题
- 使用trucate清整表空间及索引整理问题
- [整理]cygwin下使用shell时碰到的几个问题
- 使用Jmeter易遇见的问题
- 安卓极光推送的使用以及可能遇到的问题整理
- flume 使用遇到问题及解决
- JIRA使用过程出现问题整理解答
- loadrunner11使用常见问题(不断整理中)
- Unity3d使用高通Vuforia发布IOS工程遇见缺少插件问题解决
- 项目中使用百度地图遇见的问题
- Flume中同时使用Kafka Source和Kafka Sink的Topic覆盖问题
- 使用Struts2标签,s:if判断遇见问题
- 使用flume问题总结1——搭建flume+测试Syslog source
- Flume的使用问题及解决方案
- 使用tab自动补全 npm 命令中遇见的问题
- android studio使用遇见的常见问题