flume-ng负载均衡load-balance、failover集群搭建
2016-04-29 17:48
399 查看
转自:http://blog.csdn.net/morning_pig/article/details/9093149
集群采用3台机器:
[html] view
plaincopy
host1 load-balance设置
host2 机器1
host3 机器2
其中,host1 机器配置:
[html] view
plaincopy
#Define a memory channel called c1 on a1
a1.channels = c1
a1.sources = r1
a1.sinks = k1 k2
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = load_balance
a1.sinkgroups.g1.processor.selector = round_robin
a1.sinkgroups.g1.processor.backoff = true
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /tmp/flume/loadcheckpoint
a1.channels.c1.dataDirs = /tmp/flume/loaddata
a1.sources.r1.channels = c1
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 41415
a1.sinks.k1.channel = c1
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = host2
a1.sinks.k1.port = 41414
a1.sinks.k2.channel = c1
a1.sinks.k2.type = AVRO
a1.sinks.k2.hostname = host3
a1.sinks.k2.port = 41414
host2 机器配置:
[html] view
plaincopy
a2.channels = c1
a2.sources = r1
a2.sinks = k1
a2.channels.c1.type = FILE
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a2.sources.r1.channels = c1
a2.sources.r1.type = AVRO
a2.sources.r1.bind = 0.0.0.0
a2.sources.r1.port = 41414
a2.sinks.k1.channel = c1
a2.sinks.k1.type = file_roll
a2.sinks.k1.sink.directory = /tmp/load/
a2.sinks.k1.sink.rollInterval = 0
host3 机器配置:
[html] view
plaincopy
a2.channels = c1
a2.sources = r1
a2.sinks = k1
a2.channels.c1.type = FILE
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a2.sources.r1.channels = c1
a2.sources.r1.type = AVRO
a2.sources.r1.bind = 0.0.0.0
a2.sources.r1.port = 41414
a2.sinks.k1.channel = c1
a2.sinks.k1.type = file_roll
a2.sinks.k1.sink.directory = /tmp/load/
a2.sinks.k1.sink.rollInterval = 0
客户端机器暂时使用flume-ng的agent发送,配置如下:
[html] view
plaincopy
# Define a memory channel called c1 on a1
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a1.sources.r1.channels = c1
a1.sources.r1.type = exec
a1.sources.r1.command = cat /tmp/linux.log
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = host1
a1.sinks.k1.port = 41415
a1.channels = c1
a1.sources = r1
a1.sinks = k1
客户端/tmp/linux.log文件3G左右,发送给host1。
启动:
host2:
[html] view
plaincopy
bin/flume-ng agent -c conf -f conf/load-sink1.conf -n a2
host3:
[html] view
plaincopy
bin/flume-ng agent -c conf -f conf/load-sink2.conf -n a2
host1:
[html] view
plaincopy
bin/flume-ng agent -c conf -f conf/load-balance.conf -n a1
客户端:
[html] view
plaincopy
bin/flume-ng agent -c conf -f conf/client.conf -n a1
注意:启动方式最好是从下往上启动。即:先启动host2和host3,然后启动host1,最后启动client。
测试过程中,可以随时将host2或host3停止,过一段时间再启动。
这样,就测试了flume-ng的load-balance和failover功能。
集群采用3台机器:
[html] view
plaincopy
host1 load-balance设置
host2 机器1
host3 机器2
其中,host1 机器配置:
[html] view
plaincopy
#Define a memory channel called c1 on a1
a1.channels = c1
a1.sources = r1
a1.sinks = k1 k2
a1.sinkgroups = g1
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = load_balance
a1.sinkgroups.g1.processor.selector = round_robin
a1.sinkgroups.g1.processor.backoff = true
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /tmp/flume/loadcheckpoint
a1.channels.c1.dataDirs = /tmp/flume/loaddata
a1.sources.r1.channels = c1
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 41415
a1.sinks.k1.channel = c1
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = host2
a1.sinks.k1.port = 41414
a1.sinks.k2.channel = c1
a1.sinks.k2.type = AVRO
a1.sinks.k2.hostname = host3
a1.sinks.k2.port = 41414
host2 机器配置:
[html] view
plaincopy
a2.channels = c1
a2.sources = r1
a2.sinks = k1
a2.channels.c1.type = FILE
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a2.sources.r1.channels = c1
a2.sources.r1.type = AVRO
a2.sources.r1.bind = 0.0.0.0
a2.sources.r1.port = 41414
a2.sinks.k1.channel = c1
a2.sinks.k1.type = file_roll
a2.sinks.k1.sink.directory = /tmp/load/
a2.sinks.k1.sink.rollInterval = 0
host3 机器配置:
[html] view
plaincopy
a2.channels = c1
a2.sources = r1
a2.sinks = k1
a2.channels.c1.type = FILE
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a2.sources.r1.channels = c1
a2.sources.r1.type = AVRO
a2.sources.r1.bind = 0.0.0.0
a2.sources.r1.port = 41414
a2.sinks.k1.channel = c1
a2.sinks.k1.type = file_roll
a2.sinks.k1.sink.directory = /tmp/load/
a2.sinks.k1.sink.rollInterval = 0
客户端机器暂时使用flume-ng的agent发送,配置如下:
[html] view
plaincopy
# Define a memory channel called c1 on a1
a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a1.sources.r1.channels = c1
a1.sources.r1.type = exec
a1.sources.r1.command = cat /tmp/linux.log
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = host1
a1.sinks.k1.port = 41415
a1.channels = c1
a1.sources = r1
a1.sinks = k1
客户端/tmp/linux.log文件3G左右,发送给host1。
启动:
host2:
[html] view
plaincopy
bin/flume-ng agent -c conf -f conf/load-sink1.conf -n a2
host3:
[html] view
plaincopy
bin/flume-ng agent -c conf -f conf/load-sink2.conf -n a2
host1:
[html] view
plaincopy
bin/flume-ng agent -c conf -f conf/load-balance.conf -n a1
客户端:
[html] view
plaincopy
bin/flume-ng agent -c conf -f conf/client.conf -n a1
注意:启动方式最好是从下往上启动。即:先启动host2和host3,然后启动host1,最后启动client。
测试过程中,可以随时将host2或host3停止,过一段时间再启动。
这样,就测试了flume-ng的load-balance和failover功能。
相关文章推荐
- android.intent.action.MAIN与android.intent.category.LAUNCHER的区别
- CLOSE_WAIT问题
- RAID5 的那些事
- async & await 的用法
- 百度地图开发(1)-----如何从hello baidumap定位所处城市并进行切换
- jenkins email配置
- 172. Factorial Trailing Zeroes
- HDU 1789 Doing Homework again 贪心
- 关于使用thumbnailator生成缩略图异常问题
- 关于发邮件报错535 Error:authentication failed解决方法
- You raise me up
- Neutron/L3 High Availability VRRP
- 【HDU】 1532 Drainage Ditches
- count(字段)不统计null值http://blog.csdn.net/lwei_998/article/details/6112810
- FZCCHJW--GB1-0-GBpc-EUC-H:在系统中未找到字体;已替换缺少的字体。
- leetcode 11. Container With Most Water
- stitching detail输出的dot图含义
- 从零开始写聊天机器人
- AI教程!教你绘制扁平化风格的卡通人物肖像(四人组合篇)
- UVa 10976 Fractions Again?!