k8s网络之calico学习
知识准备
1.calico主要通过ipip协议与bgp协议来实现通信。前者通过ipip隧道作为通信基础,后者则是纯三层的路由交换
2.bgp协议主要由两种方式:BGP Speaker 全互联模式(node-to-node mesh)与BGP Speaker RR模式
3.本文主要探索一下calico bgp的两种模式
环境准备
组件 | 版本 |
---|---|
OS | Ubuntu 18.04.1 LTS |
docker | 18.06.0-ce |
k8s | 1.10.1 |
calico | 3.1.3 |
ip | hostname |
---|---|
192.168.56.101 | k8s-master |
192.168.56.102 | k8s-node1 |
192.168.56.103 | k8s-node2 |
192.168.56.104 | k8s-node3 |
安装
k8s安装
参考官网安装以及社区诸多大神的安装帖子,这里就不班门弄斧了
本文的k8s的环境打开了rbac,etcd加入了证书
calico安装
主要参考官方文档 https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/calico
1.calico rbac
kubectl apply -f \ https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/rbac.yaml
2.下载calico.yaml
curl \ https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calico.yaml \ -O
3.填入etcd地址
ETCD_ENDPOINTS="https://192.168.56.101:2379" sed -i "s#.*etcd_endpoints:.*# etcd_endpoints: \"${ETCD_ENDPOINTS}\"#g" calico.yaml sed -i "s#__ETCD_ENDPOINTS__#${ETCD_ENDPOINTS}#g" calico.yaml
4.将etcd证书信息填入。我的etcd证书在/etc/etcd/ssl下
ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'` ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'` ETCD_CA=`cat /etc/etcd/ssl/etcd-root-ca.pem | base64 | tr -d '\n'` sed -i "s#.*etcd-cert:.*# etcd-cert: ${ETCD_CERT}#g" calico.yaml sed -i "s#.*etcd-key:.*# etcd-key: ${ETCD_KEY}#g" calico.yaml sed -i "s#.*etcd-ca:.*# etcd-ca: ${ETCD_CA}#g" calico.yaml sed -i 's#.*etcd_ca:.*# etcd_ca: "/calico-secrets/etcd-ca"#g' calico.yaml sed -i 's#.*etcd_cert:.*# etcd_cert: "/calico-secrets/etcd-cert"#g' calico.yaml sed -i 's#.*etcd_key:.*# etcd_key: "/calico-secrets/etcd-key"#g' calico.yaml sed -i "s#__ETCD_KEY_FILE__#/etc/etcd/ssl/etcd-key.pem#g" calico.yaml sed -i "s#__ETCD_CERT_FILE__#/etc/etcd/ssl/etcd.pem#g" calico.yaml sed -i "s#__ETCD_CA_CERT_FILE__#/etc/etcd/ssl/etcd-root-ca.pem#g" calico.yaml sed -i "s#__KUBECONFIG_FILEPATH__#/etc/cni/net.d/calico-kubeconfig#g" calico.yaml
5.配置calico bgp 并且修改ip cidr:10.10.0.0/16
sed -i '/CALICO_IPV4POOL_IPIP/{n;s/Always/off/g}' calico.yaml sed -i '/CALICO_IPV4POOL_CIDR/{n;s/192.168.0.0/10.10.0.0/g}' calico.yaml
6.kubectl安装calico
kubectl apply -f calico.yaml
注意:因为calico-node需要获取操作系统的权限运行,所以要在apiserver、kubelet中加入--allow-privileged=true
查看一下状态:
root@k8s-master:/tmp# kubectl get pods -n kube-system -owide NAME READY STATUS RESTARTS AGE IP NODE calico-kube-controllers-98989846-b4n72 1/1 Running 0 18d 192.168.56.102 k8s-node1 calico-node-58pck 2/2 Running 0 18d 192.168.56.103 k8s-node2 calico-node-s2txw 2/2 Running 0 18d 192.168.56.101 k8s-master calico-node-svmbp 2/2 Running 0 18d 192.168.56.102 k8s-node1 ...
7.kubelet配置calico
找到kubelet的配置文件(我的环境在/etc/kubernetes/kubelet),加入 --network-plugin=cni 重启kubelet
8.测试一个pod
cat << EOF | kubectl create -f - apiVersion: v1 kind: Pod metadata: name: network-test namespace: test spec: containers: - image: busybox:latest command: - sleep - "3600" name: network-test EOF
root@k8s-master:~# kubectl -n test get pods -owide NAME READY STATUS RESTARTS AGE IP NODE network-test 1/1 Running 0 41s 10.10.169.139 k8s-node2
至此:calico安装已经完成
calicoctl使用
1.下载calicoctl
https://github.com/projectcalico/calicoctl/releases/download/v3.1.3/calicoctl-linux-amd64
2.查看当前的calico-node
root@k8s-master:/tmp# calicoctl get node NAME k8s-master k8s-node1 k8s-node2 calicoctl get node -o yaml 查看详细信息
3.查看当前的ippool
root@k8s-master:/tmp# calicoctl get ippool NAME CIDR default-ipv4-ippool 10.10.0.0/16 default-ipv6-ippool fdc6:1a69:2b39::/48
4.查看当前模式
root@k8s-master:/tmp# calicoctl node status Calico process is running. IPv4 BGP status +----------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +----------------+-------------------+-------+----------+-------------+ | 192.168.56.102 | node-to-node mesh | up | 07:39:02 | Established | | 192.168.56.103 | node-to-node mesh | up | 07:39:02 | Established | +----------------+-------------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@k8s-master:/tmp# netstat -anp | grep ESTABLISH | grep bird tcp 0 0 192.168.56.101:33029 192.168.56.102:179 ESTABLISHED 26558/bird tcp 0 0 192.168.56.101:58055 192.168.56.103:179 ESTABLISHED 26558/bird
当前运行在BGP Speaker 全互联模式(node-to-node mesh)模式,calico集群中的节点之间都会相互建立连接,用于路由交换。适合规模不大的集群中运行,一旦集群节点增大,mesh模式将形成一个巨大服务网格,连接数暴增
5.修改BGP Speaker RR模式
禁止mesh模式,配置bgpPeer
cat << EOF | calicoctl create -f - apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: false asNumber: 61234 EOF cat << EOF | calicoctl create -f - apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: bgppeer-global spec: peerIP: 192.168.56.103 asNumber: 61234 EOF
查看RR模式配置:
root@k8s-master:~# calicoctl get bgpconfig NAME LOGSEVERITY MESHENABLED ASNUMBER default Info false 61234 root@k8s-master:~# calicoctl get bgppeer NAME PEERIP NODE ASN bgppeer-global 192.168.56.103 (global) 61234
安装routereflector
docker run --privileged --net=host -d \ --name=calico-rr \ -e IP=192.168.56.104 \ -e ETCD_ENDPOINTS=https://192.168.56.101:2379 \ -v /etc/calico/ssl:/etc/calico/ssl \ -e ETCD_CA_CERT_FILE=/etc/calico/ssl/etcd-root-ca.pem \ -e ETCD_CERT_FILE=/etc/calico/ssl/etcd.pem \ -e ETCD_KEY_FILE=/etc/calico/ssl/etcd-key.pem \ calico/routereflector:v0.6.1
查看效果:
root@k8s-master:~# calicoctl node status Calico process is running. IPv4 BGP status +----------------+-----------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +----------------+-----------+-------+----------+-------------+ | 192.168.56.103 | global | up | 09:13:23 | Established | +----------------+-----------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@k8s-master:~# netstat -anp | grep ESTABLISH | grep bird tcp 0 0 192.168.56.101:179 192.168.56.103:54903 ESTABLISHED 26558/bird
每台机器都只会与rr建立一条连接,并且与rr通信即可拿到所有路由,大大减少了连接数量
至此,本文结束
在下才疏学浅,有撒汤漏水的,请各位不吝赐教...
- [k8s]docker calico网络&docker cluster-store
- 一文读懂生成对抗网络GANs(附学习资源)
- Android多媒体学习:播放网络上的视频
- 【学习笔记】CSW网络目录服务
- 网络编程学习笔记(获取所有网络接口)
- 海量增量学习神经网络 lsnn,the finaly nn
- 计算机网络--以太网学习
- Android阶段学习笔记7.21-7.24 之 网络编程
- 可视化Keras深度学习神经网络模型
- python网络学习(04)
- 网络硬件学习
- 发送报文过程的调度 (linux网络子系统学习 第十二节 )
- 神经网络学习笔记
- 学习培训笔记--网络
- 神经网络:学习(2)
- 残差网络resnet学习
- TensorFlow 深度学习框架(9)-- 经典卷积网络模型 : LeNet-5 模型 & Inception-v3 模型
- 如何部署 Calico 网络?- 每天5分钟玩转 Docker 容器技术(67)
- C++网络编程学习笔记1
- Android学习--下载网络图片