Elasticsearch冷热集群搭建
ES版本:6.2.4
集群环境:7台机器,每台部署一个master节点。其中3台部署2个hot节点,另外4台部署2个warm节点。共21个节点。
1. 挂盘
按实际情况分盘,一个机子上的2个data节点均分数据磁盘。
通过UUID挂盘,以防止以后换盘,盘符移动(这里拿3个盘举例)。
mkdir -p /data01a/esat mkdir -p /data02b/esat mkdir -p /data03c/esat blkid vi /etc/fstab UUID="f78d2fb8-4b00-4f08-af69-07d533b54a85" /data01a ext4 defaults 0 3 UUID="3cd45d40-e51c-42ce-9b6b-8d2c315a8f9d" /data02b ext4 defaults 0 3 UUID="396e2e59-e92c-4813-a1f1-ed560731a361" /data03c ext4 defaults 0 3 mount -a
2. 创建用户,修改磁盘权限
groupadd esadmin useradd -g esadmin esadmin useradd -g esadmin esmaster useradd -g esadmin esdata chown -R esadmin:esadmin /data01a chown -R esadmin:esadmin /data02b chown -R esadmin:esadmin /data03c chmod -R 771 /data*
3. 系统设置
vi /etc/sysctl.conf vm.swappiness = 1 vm.max_map_count = 262144 sysctl -p vi /etc/security/limits.conf @esadmin - nofile 65536 @esadmin - nproc 4096 @esadmin - memlock unlimited
4.安装JDK8
rpm -i jdk-8u171-linux-x64.rpm
5. 安装ES
将ES安装文件拷贝到对应目录下。
安装master:
cd /home/esmaster
unzip elasticsearch-6.2.4.zip
修改配置:
vi elasticsearch-6.2.4/config/elasticsearch.yml
cluster.name: es-xx
node.name: master-at-xx
node.master: true node.data: false node.ingest: false search.remote.connect: false bootstrap.memory_lock: true bootstrap.system_call_filter: false network.host: xx.xx.xx.xx http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
discovery.zen.ping.unicast.hosts: ["xx.xx.xx.xx:9300", "xx.xx.xx.xx:9300", "xx.xx.xx.xx:9300", ...]
discovery.zen.minimum_master_nodes: 4
discovery.zen.fd.ping_interval: 10s
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 3
gateway.expected_master_nodes: 7
gateway.recover_after_time: 10m
gateway.recover_after_master_nodes: 4
安装ik分词器:
./elasticsearch-6.2.4/bin/elasticsearch-plugin install file:///home/esmaster/elasticsearch-analysis-ik-6.2.4.zip
chown -R esmaster:esadmin *
安装data:
cd /home/esdata
unzip elasticsearch-6.2.4.zip
修改配置:
vi elasticsearch-6.2.4/config/elasticsearch.yml
cluster.name: es-xx node.name: data-at-40 node.master: false node.data: true node.ingest: true search.remote.connect: true search.remote.node.attr: gateway node.attr.type: hot node.attr.gateway: true path.data: ["/data01a/esat", "/data02b/esat", "/data03c/esat", ...] bootstrap.memory_lock: true bootstrap.system_call_filter: false network.host: xx.xx.xx.xx http.port: 9201 transport.tcp.port: 9301 http.cors.enabled: true http.cors.allow-origin: "*"
discovery.zen.ping.unicast.hosts: ["xx.xx.xx.xx:9300", "xx.xx.xx.xx:9300", "xx.xx.xx.xx:9300", ...]
discovery.zen.minimum_master_nodes: 4
discovery.zen.fd.ping_interval: 10s
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 3
gateway.expected_master_nodes: 7
gateway.recover_after_time: 10m
gateway.recover_after_master_nodes: 4
上面是hot节点的配置(node.attr.type: hot),warm节点类似。
安装ik分词器:
./elasticsearch-6.2.4/bin/elasticsearch-plugin install file:///home/esdata/elasticsearch-analysis-ik-6.2.4.zip
chown -R esdata:esadmin *
修改JVM配置文件
vi /home/esmaster/elasticsearch-6.2.4/config/jvm.options
master 修改jvm.options -Xms10g -Xmx10g
vi /home/esdata/elasticsearch-6.2.4/config/jvm.options
data 修改jvm.options -Xms31g -Xmx31g
6 启动
./bin/elasticsearch -p pid -d
7. 启动监控工具head和Bigdesk
unzip es_tools.zip cd es_tools nohup python -m SimpleHTTPServer 9080 &
8. 设置索引模板
put _template/app_template { "order": 0, "template": "app_*", "settings": { "index": { "unassigned": { "node_left": { "delayed_timeout": "10m" } }, "routing": { "allocation": { "include": {"type": "hot"} } }, "refresh_interval": "5s", "number_of_shards": "1", "number_of_replicas": "1" } }, "mappings": { "_doc": { "date_detection": false, "properties": { "timestamp": { "format": "yyyy-MM-dd HH:mm:ss:SSS||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss||dateOptionalTime", "type": "date" }, "@timestamp": { "format": "yyyy-MM-dd HH:mm:ss:SSS||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss||dateOptionalTime", "type": "date" }, "attimestamp": { "format": "yyyy-MM-dd HH:mm:ss:SSS||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss||dateOptionalTime", "type": "date" }, "ip": { "type": "ip" }, "message": { "search_analyzer": "ik_max_word", "analyzer": "ik_max_word", "type": "text" } }, "dynamic_templates": [ { "strings_not_analyzed": { "match_mapping_type": "string", "mapping": { "type": "keyword" } } } ] } } }
以上设置所有以“app_”命名的索引的template,默认分配到hot节点。
9.迁移索引
动态修改某个索引到warm节点。如设置"app_index"这个索引到warm节点:
PUT app_index/_settings { "index": { "routing": { "allocation": { "include": {"type": "warm"} } } } }
参考:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-cross-cluster-search.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html
- Elasticsearch集群的简单搭建
- ElasticSearch — 集群搭建
- ElasticSearch — 集群搭建
- Elasticsearch5.2.2集群搭建
- ElasticSearch集群搭建及插件安装
- ElasticSearch集群搭建及插件安装
- ElasticSearch集群搭建实例
- elasticsearch-2集群搭建的坑
- elasticsearch集群搭建实例
- elasticsearch系列八:ES 集群管理(集群规划、集群搭建、集群管理)
- 【Elasticsearch全文搜索引擎实战】之集群搭建及配置
- Linux下elasticsearch集群搭建
- Elasticsearch 初探及window下集群的搭建
- elasticsearch 6.0.0(多节点安装) + kibana 6.0.0集群环境搭建
- ElasticSearch集群搭建及插件安装
- elasticsearch 搭建-集群-java应用
- ElasticSearch的基本用法与集群搭建
- 搭建Elasticsearch 6.1分布式集群
- 使用elasticsearch搭建集群
- elasticsearch集群搭建手册