flume报错:Sinks are likely not keeping up with sources, or the buffer size is too tight
2017-03-04 15:39
483 查看
使用flume收集日志,在agent的日志中发现如下错误信息:
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
... 28 more
解决方法:
1)修改flume-ng agen的堆内存:
vim flume-env.sh
export JAVA_OPTS="-Xms1024m -Xmx1024m"默认是20m
2)修改channel的配置信息:
agent1.channels.mc1.capacity = 1000
agent1.channels.mc1.keep-alive = 60
解释:
keep-alive"这个参数用来控制channel满时影响source的发送,channel空时影响sink的消费,就是等待时间,默认是3s,超过这个时间就甩异常,一般不需配置,但是有些情况很有用,比如你得场景是每分钟开头集中发一次数据,这时每分钟的开头量可能比较大,后面会越来越小,这时你可以调大这个参数,不至于出现channel满了得情况;
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
... 28 more
解决方法:
1)修改flume-ng agen的堆内存:
vim flume-env.sh
export JAVA_OPTS="-Xms1024m -Xmx1024m"默认是20m
2)修改channel的配置信息:
agent1.channels.mc1.capacity = 1000
agent1.channels.mc1.keep-alive = 60
解释:
channels | - | |
type | - | 组件类型名称必须是memory |
capacity | 100 | 存储在 Channel 当中的最大 events 数 |
transactionCapacity | 100 | 同时刻从Source 获取,或发送到 Sink 的最大 events 数 |
keep-alive | 3 | 添加或删除一个 event 超时的秒数 |
相关文章推荐
- Flume启动运行时报错org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tight解决办法(图文详解)
- anaconda2下面安装opencv2.4.13.4完成----解决默认安装的问题----Thefunction is not implemented. Rebuild the library with Windows, GTK+ 2.x orCarbon support. If you are on Ubuntu or Debian, install libgtk2.0‑dev and pkg
- eclipse中,项目有红叉之-Project configuration is not up-to-date with pom.xml. Select: Maven->Update Project... from the project context menu or use Quick Fix
- fs: re-evaluating native module sources is not supported. If you are using the graceful-fs module
- Project configuration is not up-to-date with pom.xml. Run Maven->Update Project or use Quick Fix
- Project configuration is not up-to-date with pom.xml. Run Maven->Update Project or use Quick Fix
- The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support.
- Project configuration is not up-to-date with pom.xml. Run Maven->Update Project or use Quick Fix.
- rman在nfs文件系统上备份报错(NFS file system where the file is created or resides is not mounted with correct )
- the attribute buffer size is too small 解决方法
- 关于The function size must be used with a prefix when a default namespace is not specified解决
- Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. You
- WWW or No WWW is Not the Same! Fix with Htaccess 301 Redirect
- 错误: Some part of your SQL statement is nested too deeply. Rewrite the query or break it up into smaller queries.
- IOS 从网上下载的项目,运行时出现“The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update y”
- Cannot connect to WMI provider.You do not have permission or the server is unreachable.Note that you can only manager SQL Server 2005 and later version with SQL Server Configuration Manager.Invalid namespace [0x8004100e]
- ORA-27054: NFS file system where the file is created or resides is not mounted with correct options
- Data for Source Column 3(’Col3’) is too large for the specified buffer size.
- ORA-27054: NFS file system where the file is created or resides is not mounted with correct options
- Project configuration is not up-to-date with pom.xml. Run Maven->Update Project or use Quick Fix