用flume提交文件到hdfs系统,并保持原来的文件名信息
2016-01-05 09:10
465 查看
用flume提交文件到hdfs系统,并保持原来的文件名信息的配置方式:
# 配置files_source
# spooling
agent1.sources.files_source.type = spooldir
agent1.sources.files_source.spoolDir = /tmp/flumetest/
agent1.sources.files_source.fileHeader = true
agent1.sources.files_source.fileHeaderKey = fileName
# 配置hdfs_sink
agent1.sinks.hdfs_sink.type = hdfs
agent1.sinks.hdfs_sink.hdfs.path = hdfs://localhost:9000/myflumetestcluster/flume
agent1.sinks.hdfs_sink.fileType = SequenceFile
# agent1.sinks.hdfs_sink.fileType = DataStream #值DataStream表示文件类型,不会被压缩
# agent1.sinks.hdfs_sink.writeFormat = Text #针对DataStream,使用Text输出格式
agent1.sinks.hdfs_sink.hdfs.filePrefix = %{fileName} #把原来的文件名作为前缀
# 配置files_source
# spooling
agent1.sources.files_source.type = spooldir
agent1.sources.files_source.spoolDir = /tmp/flumetest/
agent1.sources.files_source.fileHeader = true
agent1.sources.files_source.fileHeaderKey = fileName
# 配置hdfs_sink
agent1.sinks.hdfs_sink.type = hdfs
agent1.sinks.hdfs_sink.hdfs.path = hdfs://localhost:9000/myflumetestcluster/flume
agent1.sinks.hdfs_sink.fileType = SequenceFile
# agent1.sinks.hdfs_sink.fileType = DataStream #值DataStream表示文件类型,不会被压缩
# agent1.sinks.hdfs_sink.writeFormat = Text #针对DataStream,使用Text输出格式
agent1.sinks.hdfs_sink.hdfs.filePrefix = %{fileName} #把原来的文件名作为前缀
相关文章推荐
- hadoop的hdfs文件操作实现上传文件到hdfs
- Flume环境部署和配置详解及案例大全
- java连接hdfs ha和调用mapreduce jar示例
- java实现将ftp和http的文件直接传送到hdfs
- 在Hadoop2.5.0下利用Java读写HDFS
- HDFS 文件操作
- Spark中将对象序列化存储到hdfs
- 读<王垠:一种新的操作系统设计>
- Play! Akka Flume实现的完整数据收集
- hadoop中RPC通信文件上传原理
- 测试Hadoop的hdfs的问题?
- flume自定义Interceptor
- 高可用,完全分布式Hadoop集群HDFS和MapReduce安装配置指南
- hadoop特性讲解
- HDFS 恢复某时刻删除的文件
- #Note# Analyzing Twitter Data with Apache Hadoo...
- tachyon与hdfs,以及spark整合
- CentOS6.5安装Hadoop2.7.1
- flume、kafka、storm常用命令
- Hadoop HDFS Java API