kafka 配置文件详解:producer.properties
2017-08-17 14:01
375 查看
Producer作为kafak核心组件之一,学习和分析它是很有必要的。producer配置文件在kafak的config目录下,配置好它
在以后的学习和工作中都可以起到事半功倍的效果。
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.producer.ProducerConfig for more details
############################# Producer Basics #############################
# list of brokers used for bootstrapping knowledge about the rest of the cluster
# format: host1:port1,host2:port2 ...
#指定节点列表(必填,集群需要写各个节点,我这里是单机)
metadata.broker.list=localhost:9092
# name of the partitioner class for partitioning events; default partition spreads data randomly
#指定分区处理类。默认kafka.producer.DefaultPartitioner
partitioner.class=kafka.producer.DefaultPartitioner
# specifies whether the messages are sent asynchronously (async) or synchronously (sync)
#sync同步(默认),async异步可以提高发送吞吐量
producer.type=sync
# specify the compression codec for all data generated: none, gzip, snappy, lz4.
# the old config values work as well: 0, 1, 2, 3 for none, gzip, snappy, lz4, respectively
#是否压缩,0代表不压缩,1代表用gzip压缩,2代表用snappy压缩
compression.codec=0
# message encoder
#指定序列化处理类
serializer.class=kafka.serializer.DefaultEncoder
# allow topic level compression
#如果要压缩消息,这里指定哪些topic要压缩消息,默认是empty,表示不压缩
#compressed.topics=
#设置发送数据是否需要服务端的反馈,有三个值0,1,-1
# 0:producer不会等待broker发送ack
# 1:当leader接收到消息后发送ack
# -1:当所有的follower都同步消息成功后发送ack
request.required.acks=0
############################# Async Producer #############################
# maximum time, in milliseconds, for buffering data on the producer queue
#在async模式下,当message缓存超时后,将会批量发送给broker,默认5000ms
queue.buffering.max.ms=5000
# the maximum size of the blocking queue for buffering on the producer
#在async模式下,Producer端允许buffer的最大消息量
queue.buffering.max.messages=20000
#在向producer发送ack之前,broker均需等待的最大时间
request.timeout.ms=10000
# Timeout for event enqueue:
# 0: events will be enqueued immediately or dropped if the queue is full
# -ve: enqueue will block indefinitely if the queue is full
# +ve: enqueue will block up to this many milliseconds if the queue is full
#当消息在producer端沉积的条数达到“queue.buffering.max.messages"后
#阻塞一定时间后,队列仍然没有enqueue(producer仍然没有发送出任何消息)
#此时producer可以继续阻塞,或者将消息抛弃
# -1:无阻塞超时限制,消息不会被抛弃
# 0 :立即清空队列,消息被抛弃
queue.enqueue.timeout.ms=-1
# the number of messages batched at the producer
#在async模式下,指定每次批量发送的数据量,默认200
batch.num.messages=200
补充:producer.properties的详细配置
在以后的学习和工作中都可以起到事半功倍的效果。
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.producer.ProducerConfig for more details
############################# Producer Basics #############################
# list of brokers used for bootstrapping knowledge about the rest of the cluster
# format: host1:port1,host2:port2 ...
#指定节点列表(必填,集群需要写各个节点,我这里是单机)
metadata.broker.list=localhost:9092
# name of the partitioner class for partitioning events; default partition spreads data randomly
#指定分区处理类。默认kafka.producer.DefaultPartitioner
partitioner.class=kafka.producer.DefaultPartitioner
# specifies whether the messages are sent asynchronously (async) or synchronously (sync)
#sync同步(默认),async异步可以提高发送吞吐量
producer.type=sync
# specify the compression codec for all data generated: none, gzip, snappy, lz4.
# the old config values work as well: 0, 1, 2, 3 for none, gzip, snappy, lz4, respectively
#是否压缩,0代表不压缩,1代表用gzip压缩,2代表用snappy压缩
compression.codec=0
# message encoder
#指定序列化处理类
serializer.class=kafka.serializer.DefaultEncoder
# allow topic level compression
#如果要压缩消息,这里指定哪些topic要压缩消息,默认是empty,表示不压缩
#compressed.topics=
#设置发送数据是否需要服务端的反馈,有三个值0,1,-1
# 0:producer不会等待broker发送ack
# 1:当leader接收到消息后发送ack
# -1:当所有的follower都同步消息成功后发送ack
request.required.acks=0
############################# Async Producer #############################
# maximum time, in milliseconds, for buffering data on the producer queue
#在async模式下,当message缓存超时后,将会批量发送给broker,默认5000ms
queue.buffering.max.ms=5000
# the maximum size of the blocking queue for buffering on the producer
#在async模式下,Producer端允许buffer的最大消息量
queue.buffering.max.messages=20000
#在向producer发送ack之前,broker均需等待的最大时间
request.timeout.ms=10000
# Timeout for event enqueue:
# 0: events will be enqueued immediately or dropped if the queue is full
# -ve: enqueue will block indefinitely if the queue is full
# +ve: enqueue will block up to this many milliseconds if the queue is full
#当消息在producer端沉积的条数达到“queue.buffering.max.messages"后
#阻塞一定时间后,队列仍然没有enqueue(producer仍然没有发送出任何消息)
#此时producer可以继续阻塞,或者将消息抛弃
# -1:无阻塞超时限制,消息不会被抛弃
# 0 :立即清空队列,消息被抛弃
queue.enqueue.timeout.ms=-1
# the number of messages batched at the producer
#在async模式下,指定每次批量发送的数据量,默认200
batch.num.messages=200
补充:producer.properties的详细配置
属性 | 默认值 | 描述 |
---|---|---|
metadata.broker.list | 启动时producer查询brokers的列表,可以是集群中所有brokers的一个子集。注意,这个参数只是用来获取topic的元信息用,producer会从元信息中挑选合适的broker并与之建立socket连接。格式是:host1:port1,host2:port2。 | |
request.required.acks | 0 | 参见3.2节介绍 |
request.timeout.ms | 10000 | Broker等待ack的超时时间,若等待时间超过此值,会返回客户端错误信息。 |
producer.type | sync | 同步异步模式。async表示异步,sync表示同步。如果设置成异步模式,可以允许生产者以batch的形式push数据,这样会极大的提高broker性能,推荐设置为异步。 |
serializer.class | kafka.serializer.DefaultEncoder | 序列号类,.默认序列化成 byte[] 。 |
key.serializer.class | Key的序列化类,默认同上。 | |
partitioner.class | kafka.producer.DefaultPartitioner | Partition类,默认对key进行hash。 |
compression.codec | none | 指定producer消息的压缩格式,可选参数为: “none”, “gzip” and “snappy”。关于压缩参见4.1节 |
compressed.topics | null | 启用压缩的topic名称。若上面参数选择了一个压缩格式,那么压缩仅对本参数指定的topic有效,若本参数为空,则对所有topic有效。 |
message.send.max.retries | 3 | Producer发送失败时重试次数。若网络出现问题,可能会导致不断重试。 |
retry.backoff.ms | 100 | Before each retry, the producer refreshes the metadata of relevant topics to see if a new leader has been elected. Since leader election takes a bit of time, this property specifies the amount of time that the producer waits before refreshing the metadata. |
topic.metadata.refresh.interval.ms | 600 * 1000 | The producer generally refreshes the topic metadata from brokers when there is a failure (partition missing, leader not available…). It will also poll regularly (default: every 10min so 600000ms). If you set this to a negative value, metadata will only get refreshed on failure. If you set this to zero, the metadata will get refreshed after each message sent (not recommended). Important note: the refresh happen only AFTER the message is sent, so if the producer never sends a message the metadata is never refreshed |
queue.buffering.max.ms | 5000 | 启用异步模式时,producer缓存消息的时间。比如我们设置成1000时,它会缓存1秒的数据再一次发送出去,这样可以极大的增加broker吞吐量,但也会造成时效性的降低。 |
queue.buffering.max.messages | 10000 | 采用异步模式时producer buffer 队列里最大缓存的消息数量,如果超过这个数值,producer就会阻塞或者丢掉消息。 |
queue.enqueue.timeout.ms | -1 | 当达到上面参数值时producer阻塞等待的时间。如果值设置为0,buffer队列满时producer不会阻塞,消息直接被丢掉。若值设置为-1,producer会被阻塞,不会丢消息。 |
batch.num.messages | 200 | 采用异步模式时,一个batch缓存的消息数量。达到这个数量值时producer才会发送消息。 |
send.buffer.bytes | 100 * 1024 | Socket write buffer size |
client.id | “” | The client id is a user-specified string sent in each request to help trace calls. It should logically identify the application making the request. |
相关文章推荐
- kafka配置文件详解之:producer.properties
- kafka配置文件详解:consumer.properties
- Kafka配置文件详解之:consumer.properties
- kafka配置文件详解:server.properties
- kafka配置文件详解之:server.properties
- kafka的server.properties配置文件参考示范(图文详解)(多种方式)
- kafka配置文件详解之:server.properties
- Struts2的properties配置文件详解
- struts2中struts.properties配置文件详解(及在struts.xml文件中的配置)
- Struts2的struts.properties配置文件属性详解
- 分布式消息队列kafka系列介绍 — 配置文件详解
- struts2的struts.properties配置文件详解
- 【Struts2.0】struts2的struts.properties配置文件详解
- struts2的struts.properties配置文件详解
- struts2的struts.properties配置文件详解
- Kafka配置文件Server.properties文件的配置
- Struts struts.properties 配置文件详解
- log4CXX第二篇---配置文件(properties文件)详解
- struts2的struts.properties配置文件详解
- 详解利用Spring加载Properties配置文件