Configure a Highly Available Kubernetes / etcd Cluster with Pacemaker on Fedora
2016-06-24 17:35
1006 查看
Configure a Highly Available Kubernetes / etcd Cluster with Pacemaker on Fedora
I'm going to share some of the great work that Matt Farrellee, Rob Rati and Tim St. Clairhave done with regards to figuring out $TOPIC - they get full credit for the technical details here. It's really interesting work and I thought I'd share it with the upstream community. Not to mention it gives me an opportunity to learn how this is all set up and configured.In this configuration I will set up 5 virtual machines and one VIP:
fed-master1.example.com 192.168.123.100
fed-master2.example.com 192.168.123.101
fed-master3.example.com 192.168.123.102
fed-node1.example.com 192.168.123.103
fed-node2.example.com 192.168.123.104
fed-vip.example.com 192.168.123.105
If you are wondering how I set up this environment quickly and repetitively, check out omvfrom Purpleidea. He's a clever guy with a great dev workflow. In particular, have a look at the work he has done to put his great code into a package to make distribution easier.
In summary here, I used Vagrant, KVM and omv to build and destroy this environment. I won't go into to many details about how that all works, but feel free to ask questions in the comments if needed. My omv.yaml file is located here, this might help you get up and running quickly. Just make sure you have a Fedora 22 Vagrant box that matches the name in the file. Yup, I run it all on my laptop.
Global configuration:
Configure /etc/hosts on all nodes so that name resolution works (omv can help here)
Share SSH key from master to all other nodes
To summarize what this environment will look like and what components will be running where, I have 3 master servers which will be running the pacemaker components as well as etcd and kubernetes master node services. I have 2 nodes which will be running flanneldand the kubernetes worker node services. These 2 nodes will also be running Docker. When I'm mentioning commands below, you can assume that I want them to be run on each group of nodes, unless I specify otherwise. The overall flow of the configuration will be:
Deploy VMs
Install Software
Configure etcd
Configure flannel
Configure kubernetes
Configure pacemaker
Confirm functionality
By the time you are finished you should have a highly available Active / Passive cluster configuration running kubernetes and all the required components.
Okay, so, put on your helmet and let's get started here.
Installing Software:
Here we just need to make sure we have the appropriate packages on each node. I've listed the versions that I used for this configuration at the end of the article.
Execute the following on each master nodes:
# yum -y install etcd kubernetes-master pcs fence-agents-allExecute the following on each worker nodes:
# yum -y install kubernetes-node docker flannel
Configure etcd:
Our key value store for configuration is going to be etcd. In this case, we are creating an etcd cluster so we have a highly available deployment. The config file and script for this is on github here and here.
Create the following script (also in github) and run it from master1:
etcd0=192.168.123.100 etcd1=192.168.123.101 etcd2=192.168.123.102 INITIAL_CLUSTER="etcd0=http://$etcd0:2380,etcd1=http://$etcd1:2380,etcd2=http://$etcd2:2380" for name in etcd0 etcd1 etcd2; do ssh -t ${!name} \ sed -i -e "s#.*ETCD_NAME=.*#ETCD_NAME=$name#" \ -e "s#.*ETCD_INITIAL_ADVERTISE_PEER_URLS=.*#ETCD_INITIAL_ADVERTISE_PEER_URLS=http://${!name}:2380#" \ -e "s#.*ETCD_LISTEN_PEER_URLS=.*#ETCD_LISTEN_PEER_URLS=http://${!name}:2380#" \ -e "s#.*ETCD_LISTEN_CLIENT_URLS=.*#ETCD_LISTEN_CLIENT_URLS=http://${!name}:2379,http://127.0.0.1:2379,http://127.0.0.1:4001#" \ -e "s#.*ETCD_ADVERTISE_CLIENT_URLS=.*#ETCD_ADVERTISE_CLIENT_URLS=http://${!name}:2379#" \ -e "s#.*ETCD_INITIAL_CLUSTER=.*#ETCD_INITIAL_CLUSTER=$INITIAL_CLUSTER#" \ /etc/etcd/etcd.conf done
Execute the following on all masters:
# systemctl enable etcd; systemctl start etcd; systemctl status etcd # etcdctl cluster-health; etcdctl member list
Also, check out the /etc/etcd/etcd.conf file and journal, etc... Check that out on each master and get familiar with how etcd is configured.
Configure Flannel:
We use flannel so that container A on host A can talk to container A on host B. It provides and overlay network that the containers and kubernetes can take advantage of. Oh, and it's really easy to configure. An example /etc/sysconfig/flanneld config file is on my githubrepo.
Execute the following on the worker nodes:
# echo FLANNEL_ETCD="http://192.168.123.100:2379,http://192.168.123.101:2379,http://192.168.123.102:2379" >> /etc/sysconfig/flanneld # systemctl enable flanneld; systemctl start flanneld; systemctl status flanneld # systemctl enable docker; systemctl start docker # reboot
When the servers come back up, confirm that the flannel and docker interfaces are on the same subnet.
Configure kubernetes:
Kubernetes will be our container orchestration layer. I wont' get to much into the details of the different kubernetes services, or even usage for that matter. I can assure you it is well documented and you might want to have a look here and here. I have posted my complete kubernetes config files here.
Execute the following on the master nodes:
# echo KUBE_API_ADDRESS=--address=0.0.0.0 >> /etc/kubernetes/apiserver
You can see my kubernetes master config files here.
Execute the following on the worker nodes:
# echo KUBE_MASTER=”--master=192.168.123.105:8080” >> /etc/kubernetes/config # echo KUBELET_ADDRESS=”--address=0.0.0.0” >> /etc/kubernetes/kubelet # echo KUBELET_HOSTNAME= >> /etc/kubernetes/kubelet # echo KUBELET_ARGS=”--register-node=true” >> /etc/kubernetes/kubeletKeep in mind here that the .105 address is the VIP listed in the table at the beginning of the article.
In addition, on the kubelet, you'll want to comment out the line for KUBELET_HOSTNAME, so that when it checks in with the master, it uses it's true hostname.
You can see my kubernetes node config files here.
Configure Pacemaker:
Pacemaker is going to provide our HA mechanism. You can find more information about configuring Pacemaker on the Clusters from Scratch page of their website. My /etc/corosync/corosync.conf file is posted on github here.
Execute the following on all masters:
This command will set the password for the hacluster user in order for cluster auth to function properly.
# echo hacluster | passwd -f --stdin haclusterExecute the following on master1:
# pcs cluster auth -u hacluster -p hacluster fed-master1.example.com fed-master2.example.com fed-master3.example.com # pcs cluster setup --start --name high-availability-kubernetes fed-master1.example.com fed-master2.example.com fed-master3.example.com # pcs resource create virtual-ip IPaddr2 ip=192.168.123.105 --group master # pcs resource create apiserver systemd:kube-apiserver --group master # pcs resource create scheduler systemd:kube-scheduler --group master # pcs resource create controller systemd:kube-controller-manager --group master # pcs property set stonith-enabled=falseCheck the status of the cluster:
# pcs status # pcs cluster auth
Confirm functionality:
Here we'll want to make sure everything is working.
You can check that kubernetes is functioning by making a call to the VIP, which will point to the active instance of the kubernetes API server.
Execute the following on any master node:
# kubectl -s http://192.168.123.105:8080 get nodes NAME LABELS STATUS fed-node1 kubernetes.io/hostname=fed-node1 Ready fed-node2 kubernetes.io/hostname=fed-node2 Ready
Execute the following on any master node:
# pcs status Cluster name: high-availability-kubernetes Last updated: Wed Jul 8 15:21:35 2015 Last change: Wed Jul 8 12:38:32 2015 Stack: corosync Current DC: fed-master1.example.com (1) - partition with quorum Version: 1.1.12-a9c8177 3 Nodes configured 4 Resources configured Online: [ fed-master1.example.com fed-master2.example.com fed-master3.example.com ] Full list of resources: Resource Group: master virtual-ip (ocf::heartbeat:IPaddr2): Started fed-master1.example.com apiserver (systemd:kube-apiserver): Started fed-master1.example.com scheduler (systemd:kube-scheduler): Started fed-master1.example.com controller (systemd:kube-controller-manager): Started fed-master1.example.com PCSD Status: fed-master1.example.com: Online fed-master2.example.com: Online fed-master3.example.com: OnlineDaemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabledYou can see that everything is up and running. It shows that the resource group is running on fed-master1.example.com. Well, we might as well place that in standby and make sure it starts on another node and that we can still execute kubernetes commands.
# pcs cluster standby fed-master1.example.comNow, check the resources again:
# pcs status Cluster name: high-availability-kubernetes Last updated: Wed Jul 8 15:24:17 2015 Last change: Wed Jul 8 15:23:59 2015 Stack: corosync Current DC: fed-master1.example.com (1) - partition with quorum Version: 1.1.12-a9c8177 3 Nodes configured 4 Resources configured Node fed-master1.example.com (1): standby Online: [ fed-master2.example.com fed-master3.example.com ] Full list of resources: Resource Group: master virtual-ip (ocf::heartbeat:IPaddr2): Started fed-master2.example.com apiserver (systemd:kube-apiserver): Started fed-master2.example.com scheduler (systemd:kube-scheduler): Started fed-master2.example.com controller (systemd:kube-controller-manager): Started fed-master2.example.com PCSD Status: fed-master1.example.com: Online fed-master2.example.com: Online fed-master3.example.com: Online Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabledYou can see that it moved over to fed-master2.example.com. Now, can I still get node status?
# kubectl -s http://192.168.123.105:8080 get nodes NAME LABELS STATUS fed-node1 kubernetes.io/hostname=fed-node1 Ready fed-node2 kubernetes.io/hostname=fed-node2 Ready
Yes. I can. So, enjoy. Maybe deploy some kubernetes apps?
Package versions:
This tech changes quickly, so for reference, here's what I used to set this all up.
Master nodes:
# rpm -qa selinux* kubernetes-master etcd fence-agents-all fence-agents-all-4.0.16-1.fc22.x86_64 kubernetes-master-0.19.0-0.7.gitb2e9fed.fc22.x86_64 etcd-2.0.11-2.fc22.x86_64 selinux-policy-3.13.1-128.2.fc22.noarch selinux-policy-targeted-3.13.1-128.2.fc22.noarch
Worker nodes:
# rpm -qa kubernetes-node docker flannel selinux* selinux-policy-3.13.1-128.2.fc22.noarch selinux-policy-targeted-3.13.1-128.2.fc22.noarch kubernetes-node-0.19.0-0.7.gitb2e9fed.fc22.x86_64 docker-1.6.0-3.git9d26a07.fc22.x86_64 flannel-0.2.0-7.fc22.x86_64
And that concludes this article. I hope it was helpful. Feel free to leave some comments or suggestions. It would be cool to containerize Pacemaker and get this running on a Fedora Atomic host.
原地址: http://www.colliernotes.com/2015/07/configure-highly-available-kubernetes.html
相关文章推荐
- Rabbitmq集群搭建笔记
- ZBUS高可用HA介绍
- 国内第一篇详细讲解hadoop2的automatic HA+Federation+Yarn配置的教程
- 利用pgpool实现PostgreSQL的高可用
- Kafka设计解析(三)- Kafka High Availability (下)
- How to Setup a windows 2008 Failover cluster[01]
- 非常不错的rabbitmq集群高可用部署
- JuniperFW HA
- vMotion、SvMotion、HA和FT的详细对比说明
- Kubernetes 和 flannel
- vMotion、SvMotion、HA和FT的详细对比说明
- XenMotion 与HA的区别
- mysql HA
- 企业web高可用集群实战之lvs+keepalived+mysql HA
- SRX防火墙cluster配置步骤
- eclipse错误处理
- VMware HA实战攻略之一软硬件环境准备
- VMware HA实战攻略之二实验环境准备
- VMware HA实战攻略之三添加主机及存储
- VMware HA实战攻略之四VMware HA安装及配置