作者:Tatsuya Naganawa 译者:TF编译组
集群范围内的更新是很重要的功能,可以在保证生产集群的SLA的条件下提供集群中最新功能的有效性。
由于Tungsten Fabric在MPLS-VPN中使用类似的协议,因此,根据我的尝试,即使Control的模块版本和vRouter的模块版本不同,基本的互操作性还是有的。
因此,总体思路是,首先一个一个地更新controller,然后并根据需要使用vMotion或maintence模式逐一更新vRouter。
另外,Tungsten Fabric controller还支持名为ISSU的奇特功能,但是我认为这个名称有些混乱,因为Tungsten Fabric controller与route reflector非常相似,而不是routing-engine。
• https://github.com/Juniper/contrail-specs/blob/master/contrail-issu.md
因此,基本思路是,首先将所有配置(configs)复制到新创建的controller(或route reflectors),然后更新vRouter设置(如果服务器可以重新启动,则更新vRouter模块)以使用这些新的controller。通过这个过程,vRouter模块更新的回滚操作也将更加容易。
下面我来描述一下这个过程。
就地更新(in-place update)
由于ansible-deployer遵循等幂行为(idempotent behavior),因此更新与安装没有太大区别。以下的命令将更新所有模块。
1 2 3 4 5 |
cd contrail-ansible-deployer git pull vi config/instances.yaml (update CONTRAIL_CONTAINER_TAG) ansible-playbook -e orchestrator=xxx -i inventory/ playbooks/install_contrail.yml |
一个限制是由于此命令几乎同时重启了所有节点,因此不容易做到一一重启controller和vRouter。另外,从instances.yaml中删除其它节点将不起作用,因为一个节点的更新需要其它节点的某些参数。
• 例如,vRouter更新需要控件的IP,该IP是从instances.yaml中的control role节点推导出的
为了克服这个问题,从R2005版本开始,ziu.yaml就加入了这个功能,至少对于控制平面来说,要逐一更新。.
https://github.com/Juniper/contrail-ansible-deployer/blob/master/README_ziu.md
1 2 3 4 5 6 |
cd contrail-ansible-deployer git pull vi config/instances.yaml (update CONTRAIL_CONTAINER_TAG) ansible-playbook -e orchestrator=xxx -i inventory/ playbooks/ziu.yml ansible-playbook -e orchestrator=xxx -i inventory/ playbooks/install_contrail.yml |
就我试过的情况来看,当控制平面更新时,它会进行串联更新和重启控制进程,所以没有看到掉包的现象。
在install_contrail.yaml期间,控制进程的重启是被跳过的,因为它们已经更新了
当执行vrouter-agent重启时,仍然会出现一些丢包现象,所以如果可以的话,建议进行工作负载迁移。
ISSU
即使容器格式的差异很大(例如从4.x到5.x),我们也可以使用ISSU,因为它创建了一个新的controller集群,并在其中复制数据。
首先,我来描述最简单的情况,即一个旧controller和一个新controller,以查看整个过程。所有命令都在新的controller上键入。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
old-controller: ip: 172.31.2.209 hostname: ip-172-31-2-209 new-controller: ip: 172.31.1.154 hostname: ip-172-31-1-154 (both controllers are installed with this instances.yaml) provider_config: bms: ssh_user: root ssh_public_key: /root/.ssh/id_rsa.pub ssh_private_key: /root/.ssh/id_rsa domainsuffix: local ntpserver: 0.centos.pool.ntp.org instances: bms1: provider: bms roles: config_database: config: control: analytics: analytics_database: webui: ip: x.x.x.x ## controller's ip contrail_configuration: CONTRAIL_CONTAINER_TAG: r5.1 KUBERNETES_CLUSTER_PROJECT: {} JVM_EXTRA_OPTS: "-Xms128m -Xmx1g" global_configuration: CONTAINER_REGISTRY: tungstenfabric |
[commands]
1. 停止批处理作业
1 2 3 |
docker stop config_devicemgr_1 docker stop config_schema_1 docker stop config_svcmonitor_1 |
2. 在cassandra上注册新的control,在它们之间运行bgp
1 2 |
docker exec -it config_api_1 bash python /opt/contrail/utils/provision_control.py --host_name ip-172-31-1-154.local --host_ip 172.31.1.154 --api_server_ip 172.31.2.209 --api_server_port 8082 --oper add --router_asn 64512 --ibgp_auto_mesh |
3. 在控制器之间同步数据
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
vi contrail-issu.conf (write down this) [DEFAULTS] old_rabbit_address_list = 172.31.2.209 old_rabbit_port = 5673 new_rabbit_address_list = 172.31.1.154 new_rabbit_port = 5673 old_cassandra_address_list = 172.31.2.209:9161 old_zookeeper_address_list = 172.31.2.209:2181 new_cassandra_address_list = 172.31.1.154:9161 new_zookeeper_address_list = 172.31.1.154:2181 new_api_info={"172.31.1.154": [("root"), ("password")]} ## ssh public-key can be used image_id=`docker images | awk '/config-api/{print $3}' | head -1` docker run --rm -it --network host -v $(pwd)/contrail-issu.conf:/etc/contrail/contrail-issu.conf --entrypoint /bin/bash -v /root/.ssh:/root/.ssh $image_id -c "/usr/bin/contrail-issu-pre-sync -c /etc/contrail/contrail-issu.conf" |
4. 启动进程进行实时数据同步
1 |
docker run --rm --detach -it --network host -v $(pwd)/contrail-issu.conf:/etc/contrail/contrail-issu.conf --entrypoint /bin/bash -v /root/.ssh:/root/.ssh --name issu-run-sync $image_id -c "/usr/bin/contrail-issu-run-sync -c /etc/contrail/contrail-issu.conf" |
(如果需要,请检查日志)
1 |
docker exec -t issu-run-sync tail -f /var/log/contrail/issu_contrail_run_sync.log |
5. (更新 vrouters)
6. 在结束后停止作业,同步所有数据
1 2 3 4 5 |
docker rm -f issu-run-sync image_id=`docker images | awk '/config-api/{print $3}' | head -1` docker run --rm -it --network host -v $(pwd)/contrail-issu.conf:/etc/contrail/contrail-issu.conf --entrypoint /bin/bash -v /root/.ssh:/root/.ssh --name issu-run-sync $image_id -c "/usr/bin/contrail-issu-post-sync -c /etc/contrail/contrail-issu.conf" docker run --rm -it --network host -v $(pwd)/contrail-issu.conf:/etc/contrail/contrail-issu.conf --entrypoint /bin/bash -v /root/.ssh:/root/.ssh --name issu-run-sync $image_id -c "/usr/bin/contrail-issu-zk-sync -c /etc/contrail/contrail-issu.conf" |
7. 从cassandra中去掉旧的节点,加入新的节点
1 2 3 4 5 6 7 8 9 10 11 |
vi issu.conf (write down this) [DEFAULTS] db_host_info={"172.31.1.154": "ip-172-31-1-154.local"} config_host_info={"172.31.1.154": "ip-172-31-1-154.local"} analytics_host_info={"172.31.1.154": "ip-172-31-1-154.local"} control_host_info={"172.31.1.154": "ip-172-31-1-154.local"} api_server_ip=172.31.1.154 docker cp issu.conf config_api_1:issu.conf docker exec -it config_api_1 python /opt/contrail/utils/provision_issu.py -c issu.conf |
8. 启动批处理作业
1 2 3 |
docker start config_devicemgr_1 docker start config_schema_1 docker start config_svcmonitor_1 |
以下将是可能的检查点。
1. 步骤3之后,你可以尝试使用contrail-api-cli ls -l \*来查看所有数据是否已成功复制,并且可以通过ist.py ctr nei来查看controller之间的ibgp是否已启动。
2. 步骤4之后,你可以修改旧数据库,以查看更改是否可以成功传播到新数据库。
接下来,我将在使用编排器和两个vRouter的条件下,讨论更为实际的情况。
编排器集成
为了说明结合编排器的情况,我尝试用ansible-deployer部署两个vRouters和kubernetes。
即使与编排器结合使用,总体过程也不会有太大不同。
需要注意的是,需要在什么时候将kube-manager更改为新的。
从某种意义上讲,由于kube-manager动态订阅了来自kube-apiserver的事件并更新了Tungsten Fabric配置数据库(config-database),因此它类似于批处理作业,例如schema-transformer、svc-monitor和device-manager。因此,我使用此类批处理作业,同时停止并启动了旧的或新的kube-manager(实际上也包括webui),但是可能需要根据每个设置进行更改。
这个示例中的总体过程如下所示。
1.设置一个controller(带有一个kube-manager和kubernetes-master)和两个vRouter
2.设置一个新的controller(带有一个kube-manager,但kubernetes-master与旧的controller相同)
3.停止批处理作业、新controller的kube-manager和webui
4.启动ISSU进程并继续执行,直到开始运行同步(run-sync)
->将在controller之间建立iBGP
5.根据新controller的ansible-ployer一个一个地更新vRouter
->将一个vRouter移至新的vRouter时,新的controller也将获得
4.启动ISSU进程并继续执行,直到开始运行同步(run-sync)
->将在controller之间建立iBGP
5.根据新controller的ansible-ployer一个一个地更新vRouter
->将一个vRouter移至新的vRouter时,新的controller也将获得
k8s-default-pod-network的路由目标(route-target),并且容器之间仍然可以ping通正常工作(ist.py ctr路由摘要以及ping的结果将在稍后附上)
6.将所有vRouter移至新的controller后,停止批处理作业、旧controller上的kube-manager和webui
之后,继续执行ISSU进程,新controller上启动批处理作业, kube-manager和webui
->这个阶段从开始到结束,你都无法手动更改config-database,因此可能需要一些维护时间
(整个过程可能会持续5至15分钟,可以ping通,但是直到启动新的kube-manager时,新容器的创建才能正常工作)
7.最后,停止旧的节点上的control、config和config-database
更新vRouters时,我使用了controller的provider: bms-maint,k8s_master和vRouter,它们都已经更改为新的,以避免由于容器重启而造成干扰。我附上了原始instances.yaml和更新vRouter的instances.yaml,以便大家获取更多详细信息。
• https://github.com/tnaganawa/tungstenfabric-docs/blob/master/instances-yaml-for-ISSU
我还将在每个阶段附加ist.py ctr nei和ist.py ctr路由摘要的结果,以说明发生的相关细节。
• 请注意,在此示例中,我实际上并未更新模块,因为此设置主要是为了突出ISSU进程(因为即使模块版本相同,ansible-deployer也会重新创建vrouter-agent容器,同时即使完成实际的模块更新,丢包的数量也不会有太大不同。)
1 2 3 |
old-controller: 172.31.19.25 new-controller: 172.31.13.9 two-vRouters: 172.31.25.102, 172.31.33.175 |
开始issu之前:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei Introspect Host: 172.31.19.25 +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | ip-172-31-25-102.local | 172.31.25.102 | 0 | XMPP | internal | Established | in sync | 0 | n/a | | ip-172-31-33-175.local | 172.31.33.175 | 0 | XMPP | internal | Established | in sync | 0 | n/a | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ [root@ip-172-31-13-9 ~]# [root@ip-172-31-13-9 ~]# [root@ip-172-31-13-9 ~]# [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei Introspect Host: 172.31.13.9 [root@ip-172-31-13-9 ~]# -> iBGP is not configured yet [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr route summary Introspect Host: 172.31.19.25 +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | name | prefixes | paths | primary_paths | secondary_paths | infeasible_paths | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | default-domain:default- | 0 | 0 | 0 | 0 | 0 | | project:__link_local__:__link_local__.inet.0 | | | | | | | default-domain:default-project:dci- | 0 | 0 | 0 | 0 | 0 | | network:__default__.inet.0 | | | | | | | default-domain:default-project:dci-network:dci- | 0 | 0 | 0 | 0 | 0 | | network.inet.0 | | | | | | | default-domain:default-project:default-virtual- | 0 | 0 | 0 | 0 | 0 | | network:default-virtual-network.inet.0 | | | | | | | inet.0 | 0 | 0 | 0 | 0 | 0 | | default-domain:default-project:ip-fabric:ip- | 7 | 7 | 2 | 5 | 0 | | fabric.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-pod-network | 7 | 7 | 4 | 3 | 0 | | :k8s-default-pod-network.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-service- | 7 | 7 | 1 | 6 | 0 | | network:k8s-default-service-network.inet.0 | | | | | | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ [root@ip-172-31-13-9 ~]# [root@ip-172-31-13-9 ~]# [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr route summary Introspect Host: 172.31.13.9 +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | name | prefixes | paths | primary_paths | secondary_paths | infeasible_paths | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | default-domain:default- | 0 | 0 | 0 | 0 | 0 | | project:__link_local__:__link_local__.inet.0 | | | | | | | default-domain:default-project:dci- | 0 | 0 | 0 | 0 | 0 | | network:__default__.inet.0 | | | | | | | default-domain:default-project:dci-network:dci- | 0 | 0 | 0 | 0 | 0 | | network.inet.0 | | | | | | | default-domain:default-project:default-virtual- | 0 | 0 | 0 | 0 | 0 | | network:default-virtual-network.inet.0 | | | | | | | inet.0 | 0 | 0 | 0 | 0 | 0 | | default-domain:default-project:ip-fabric:ip- | 0 | 0 | 0 | 0 | 0 | | fabric.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-pod-network | 0 | 0 | 0 | 0 | 0 | | :k8s-default-pod-network.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-service- | 0 | 0 | 0 | 0 | 0 | | network:k8s-default-service-network.inet.0 | | | | | | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ [root@ip-172-31-13-9 ~]# |
-> 在新的控制器里没有导入路由
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
[root@ip-172-31-19-25 contrail-ansible-deployer]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE cirros-deployment-75c98888b9-6qmcm 1/1 Running 0 4m58s 10.47.255.249 ip-172-31-25-102.ap-northeast-1.compute.internal <none> cirros-deployment-75c98888b9-lxq4k 1/1 Running 0 4m58s 10.47.255.250 ip-172-31-33-175.ap-northeast-1.compute.internal <none> [root@ip-172-31-19-25 contrail-ansible-deployer]# / # ip -o a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever 13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue \ link/ether 02:6b:dc:98:ac:95 brd ff:ff:ff:ff:ff:ff 13: eth0 inet 10.47.255.249/12 scope global eth0\ valid_lft forever preferred_lft forever / # ping 10.47.255.250 PING 10.47.255.250 (10.47.255.250): 56 data bytes 64 bytes from 10.47.255.250: seq=0 ttl=63 time=2.155 ms 64 bytes from 10.47.255.250: seq=1 ttl=63 time=0.904 ms ^C --- 10.47.255.250 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.904/1.529/2.155 ms / # |
-> 两个vRouter每个都有一个容器,在两个容器之间ping的结果正常。
在provision_control之后:
1 2 3 4 5 6 7 8 9 10 11 12 |
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei Introspect Host: 172.31.19.25 +------------------------+---------------+----------+----------+-----------+-------------+-----------------+------------+-----------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +------------------------+---------------+----------+----------+-----------+-------------+-----------------+------------+-----------+ | ip-172-31-13-9.local | 172.31.13.9 | 64512 | BGP | internal | Idle | not advertising | 0 | n/a | | ip-172-31-25-102.local | 172.31.25.102 | 0 | XMPP | internal | Established | in sync | 0 | n/a | | ip-172-31-33-175.local | 172.31.33.175 | 0 | XMPP | internal | Established | in sync | 0 | n/a | +------------------------+---------------+----------+----------+-----------+-------------+-----------------+------------+-----------+ [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei Introspect Host: 172.31.13.9 [root@ip-172-31-13-9 ~]# |
-> iBGP 在老的控制器上,但是新的控制器还没有那些配置(在执行pre-sync后,这会被复制到新的控制器上)
在run-sync之后:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei Introspect Host: 172.31.19.25 +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | ip-172-31-13-9.local | 172.31.13.9 | 64512 | BGP | internal | Established | in sync | 0 | n/a | | ip-172-31-25-102.local | 172.31.25.102 | 0 | XMPP | internal | Established | in sync | 0 | n/a | | ip-172-31-33-175.local | 172.31.33.175 | 0 | XMPP | internal | Established | in sync | 0 | n/a | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei Introspect Host: 172.31.13.9 +-----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +-----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+ | ip-172-31-19-25.local | 172.31.19.25 | 64512 | BGP | internal | Established | in sync | 0 | n/a | +-----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+ [root@ip-172-31-13-9 ~]# |
-> iBGP 建立起来了,ctr route summary没有改变,因为新的控制器没有k8s-default-pod-network的路由目标(route-target),路由目标(route target)过滤组织了导入这些前缀。
在迁移节点到新的controller之后:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
/ # ping 10.47.255.250 PING 10.47.255.250 (10.47.255.250): 56 data bytes 64 bytes from 10.47.255.250: seq=0 ttl=63 time=1.684 ms 64 bytes from 10.47.255.250: seq=1 ttl=63 time=0.835 ms 64 bytes from 10.47.255.250: seq=2 ttl=63 time=0.836 ms (snip) 64 bytes from 10.47.255.250: seq=37 ttl=63 time=0.878 ms 64 bytes from 10.47.255.250: seq=38 ttl=63 time=0.823 ms 64 bytes from 10.47.255.250: seq=39 ttl=63 time=0.820 ms 64 bytes from 10.47.255.250: seq=40 ttl=63 time=1.364 ms 64 bytes from 10.47.255.250: seq=44 ttl=63 time=2.209 ms 64 bytes from 10.47.255.250: seq=45 ttl=63 time=0.869 ms 64 bytes from 10.47.255.250: seq=46 ttl=63 time=0.857 ms 64 bytes from 10.47.255.250: seq=47 ttl=63 time=0.855 ms 64 bytes from 10.47.255.250: seq=48 ttl=63 time=0.845 ms 64 bytes from 10.47.255.250: seq=49 ttl=63 time=0.842 ms 64 bytes from 10.47.255.250: seq=50 ttl=63 time=0.885 ms 64 bytes from 10.47.255.250: seq=51 ttl=63 time=0.891 ms 64 bytes from 10.47.255.250: seq=52 ttl=63 time=0.909 ms 64 bytes from 10.47.255.250: seq=53 ttl=63 time=0.867 ms 64 bytes from 10.47.255.250: seq=54 ttl=63 time=0.884 ms 64 bytes from 10.47.255.250: seq=55 ttl=63 time=0.865 ms 64 bytes from 10.47.255.250: seq=56 ttl=63 time=0.840 ms 64 bytes from 10.47.255.250: seq=57 ttl=63 time=0.877 ms ^C --- 10.47.255.250 ping statistics --- 58 packets transmitted, 55 packets received, 5% packet loss round-trip min/avg/max = 0.810/0.930/2.209 ms / # |
-> 在vrouter-agent 重启后,可以看到丢了3个包(序号40-44)。在迁移vRouter到新的之后,ping工作地很好。
1 |
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei |
检查主机:172.31.19.25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | ip-172-31-13-9.local | 172.31.13.9 | 64512 | BGP | internal | Established | in sync | 0 | n/a | | ip-172-31-33-175.local | 172.31.33.175 | 0 | XMPP | internal | Established | in sync | 0 | n/a | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei Introspect Host: 172.31.13.9 +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | ip-172-31-19-25.local | 172.31.19.25 | 64512 | BGP | internal | Established | in sync | 0 | n/a | | ip-172-31-25-102.local | 172.31.25.102 | 0 | XMPP | internal | Established | in sync | 0 | n/a | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ [root@ip-172-31-13-9 ~]# |
-> 两个控制器具有XMPP连接,建立了IBGP
1 |
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr route summary |
检查主机: 172.31.19.25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | name | prefixes | paths | primary_paths | secondary_paths | infeasible_paths | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | default-domain:default- | 0 | 0 | 0 | 0 | 0 | | project:__link_local__:__link_local__.inet.0 | | | | | | | default-domain:default-project:dci- | 0 | 0 | 0 | 0 | 0 | | network:__default__.inet.0 | | | | | | | default-domain:default-project:dci-network:dci- | 0 | 0 | 0 | 0 | 0 | | network.inet.0 | | | | | | | default-domain:default-project:default-virtual- | 0 | 0 | 0 | 0 | 0 | | network:default-virtual-network.inet.0 | | | | | | | inet.0 | 0 | 0 | 0 | 0 | 0 | | default-domain:default-project:ip-fabric:ip- | 7 | 7 | 1 | 6 | 0 | | fabric.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-pod-network | 7 | 7 | 1 | 6 | 0 | | :k8s-default-pod-network.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-service- | 7 | 7 | 0 | 7 | 0 | | network:k8s-default-service-network.inet.0 | | | | | | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr route summary |
检查主机: 172.31.13.9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | name | prefixes | paths | primary_paths | secondary_paths | infeasible_paths | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | default-domain:default- | 0 | 0 | 0 | 0 | 0 | | project:__link_local__:__link_local__.inet.0 | | | | | | | default-domain:default-project:dci- | 0 | 0 | 0 | 0 | 0 | | network:__default__.inet.0 | | | | | | | default-domain:default-project:dci-network:dci- | 0 | 0 | 0 | 0 | 0 | | network.inet.0 | | | | | | | default-domain:default-project:default-virtual- | 0 | 0 | 0 | 0 | 0 | | network:default-virtual-network.inet.0 | | | | | | | inet.0 | 0 | 0 | 0 | 0 | 0 | | default-domain:default-project:ip-fabric:ip- | 7 | 7 | 1 | 6 | 0 | | fabric.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-pod-network | 7 | 7 | 3 | 4 | 0 | | :k8s-default-pod-network.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-service- | 7 | 7 | 1 | 6 | 0 | | network:k8s-default-service-network.inet.0 | | | | | | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ [root@ip-172-31-13-9 ~]# |
-> 因为两个控制器具有至少一个容器来自k8s-default-pod-network, 它们使用iBGP来交换前缀,因此它们具有同一个前缀
在将第二个vrouter迁移到新的控制器之后:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
/ # ping 10.47.255.250 PING 10.47.255.250 (10.47.255.250): 56 data bytes 64 bytes from 10.47.255.250: seq=0 ttl=63 time=1.750 ms 64 bytes from 10.47.255.250: seq=1 ttl=63 time=0.815 ms 64 bytes from 10.47.255.250: seq=2 ttl=63 time=0.851 ms 64 bytes from 10.47.255.250: seq=3 ttl=63 time=0.809 ms (snip) 64 bytes from 10.47.255.250: seq=34 ttl=63 time=0.853 ms 64 bytes from 10.47.255.250: seq=35 ttl=63 time=0.848 ms 64 bytes from 10.47.255.250: seq=36 ttl=63 time=0.833 ms 64 bytes from 10.47.255.250: seq=37 ttl=63 time=0.832 ms 64 bytes from 10.47.255.250: seq=38 ttl=63 time=0.910 ms 64 bytes from 10.47.255.250: seq=42 ttl=63 time=2.071 ms 64 bytes from 10.47.255.250: seq=43 ttl=63 time=0.826 ms 64 bytes from 10.47.255.250: seq=44 ttl=63 time=0.853 ms 64 bytes from 10.47.255.250: seq=45 ttl=63 time=0.851 ms 64 bytes from 10.47.255.250: seq=46 ttl=63 time=0.853 ms 64 bytes from 10.47.255.250: seq=47 ttl=63 time=0.851 ms 64 bytes from 10.47.255.250: seq=48 ttl=63 time=0.855 ms 64 bytes from 10.47.255.250: seq=49 ttl=63 time=0.869 ms 64 bytes from 10.47.255.250: seq=50 ttl=63 time=0.833 ms 64 bytes from 10.47.255.250: seq=51 ttl=63 time=0.859 ms 64 bytes from 10.47.255.250: seq=52 ttl=63 time=0.866 ms 64 bytes from 10.47.255.250: seq=53 ttl=63 time=0.840 ms 64 bytes from 10.47.255.250: seq=54 ttl=63 time=0.841 ms 64 bytes from 10.47.255.250: seq=55 ttl=63 time=0.854 ms ^C --- 10.47.255.250 ping statistics --- 56 packets transmitted, 53 packets received, 5% packet loss round-trip min/avg/max = 0.799/0.888/2.071 ms / # -> 3 packet loss is seen (seq 38-42) [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei Introspect Host: 172.31.19.25 +----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+ | ip-172-31-13-9.local | 172.31.13.9 | 64512 | BGP | internal | Established | in sync | 0 | n/a | +----------------------+--------------+----------+----------+-----------+-------------+------------+------------+-----------+ [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei Introspect Host: 172.31.13.9 +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | ip-172-31-19-25.local | 172.31.19.25 | 64512 | BGP | internal | Established | in sync | 0 | n/a | | ip-172-31-25-102.local | 172.31.25.102 | 0 | XMPP | internal | Established | in sync | 0 | n/a | | ip-172-31-33-175.local | 172.31.33.175 | 0 | XMPP | internal | Established | in sync | 0 | n/a | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ [root@ip-172-31-13-9 ~]# |
-> 新的控制器具有两个XMPP连接。
1 |
[root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr route summary |
检查主机:172.31.19.25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | name | prefixes | paths | primary_paths | secondary_paths | infeasible_paths | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | default-domain:default- | 0 | 0 | 0 | 0 | 0 | | project:__link_local__:__link_local__.inet.0 | | | | | | | default-domain:default-project:dci- | 0 | 0 | 0 | 0 | 0 | | network:__default__.inet.0 | | | | | | | default-domain:default-project:dci-network:dci- | 0 | 0 | 0 | 0 | 0 | | network.inet.0 | | | | | | | default-domain:default-project:default-virtual- | 0 | 0 | 0 | 0 | 0 | | network:default-virtual-network.inet.0 | | | | | | | inet.0 | 0 | 0 | 0 | 0 | 0 | | default-domain:default-project:ip-fabric:ip- | 0 | 0 | 0 | 0 | 0 | | fabric.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-pod-network | 0 | 0 | 0 | 0 | 0 | | :k8s-default-pod-network.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-service- | 0 | 0 | 0 | 0 | 0 | | network:k8s-default-service-network.inet.0 | | | | | | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr route summary |
检查主机:172.31.13.9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | name | prefixes | paths | primary_paths | secondary_paths | infeasible_paths | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ | default-domain:default- | 0 | 0 | 0 | 0 | 0 | | project:__link_local__:__link_local__.inet.0 | | | | | | | default-domain:default-project:dci- | 0 | 0 | 0 | 0 | 0 | | network:__default__.inet.0 | | | | | | | default-domain:default-project:dci-network:dci- | 0 | 0 | 0 | 0 | 0 | | network.inet.0 | | | | | | | default-domain:default-project:default-virtual- | 0 | 0 | 0 | 0 | 0 | | network:default-virtual-network.inet.0 | | | | | | | inet.0 | 0 | 0 | 0 | 0 | 0 | | default-domain:default-project:ip-fabric:ip- | 7 | 7 | 2 | 5 | 0 | | fabric.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-pod-network | 7 | 7 | 4 | 3 | 0 | | :k8s-default-pod-network.inet.0 | | | | | | | default-domain:k8s-default:k8s-default-service- | 7 | 7 | 1 | 6 | 0 | | network:k8s-default-service-network.inet.0 | | | | | | +----------------------------------------------------+----------+-------+---------------+-----------------+------------------+ [root@ip-172-31-13-9 ~]# |
-> 老的控制器不再具有前缀。
在ISSU过程结束后,新的kube-manager 启动:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
[root@ip-172-31-19-25 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE cirros-deployment-75c98888b9-6qmcm 1/1 Running 0 34m 10.47.255.249 ip-172-31-25-102.ap-northeast-1.compute.internal <none> cirros-deployment-75c98888b9-lxq4k 1/1 Running 0 34m 10.47.255.250 ip-172-31-33-175.ap-northeast-1.compute.internal <none> cirros-deployment2-648b98685f-b8pxw 1/1 Running 0 15s 10.47.255.247 ip-172-31-25-102.ap-northeast-1.compute.internal <none> cirros-deployment2-648b98685f-nv7z9 1/1 Running 0 15s 10.47.255.248 ip-172-31-33-175.ap-northeast-1.compute.internal <none> [root@ip-172-31-19-25 ~]# -> 通过新的IP创建容器(10.47.255.247, 10.47.255.248 是来自新控制器的新地址) [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.19.25 ctr nei Introspect Host: 172.31.19.25 +----------------------+--------------+----------+----------+-----------+--------+-----------------+------------+-----------------------------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +----------------------+--------------+----------+----------+-----------+--------+-----------------+------------+-----------------------------+ | ip-172-31-13-9.local | 172.31.13.9 | 64512 | BGP | internal | Active | not advertising | 1 | 2019-Jun-23 05:37:02.614003 | +----------------------+--------------+----------+----------+-----------+--------+-----------------+------------+-----------------------------+ [root@ip-172-31-13-9 ~]# ./contrail-introspect-cli/ist.py --host 172.31.13.9 ctr nei Introspect Host: 172.31.13.9 +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ | ip-172-31-25-102.local | 172.31.25.102 | 0 | XMPP | internal | Established | in sync | 0 | n/a | | ip-172-31-33-175.local | 172.31.33.175 | 0 | XMPP | internal | Established | in sync | 0 | n/a | +------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+ [root@ip-172-31-13-9 ~]# |
-> 新控制器不再有iBGP路由到旧的控制器。旧控制器依然具有iBGP路由条目,虽然这个过程很快就要停止:)
在停止了就的控制器后,配置:
1 2 3 4 5 6 7 8 9 |
[root@ip-172-31-19-25 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE cirros-deployment-75c98888b9-6qmcm 1/1 Running 0 48m 10.47.255.249 ip-172-31-25-102.ap-northeast-1.compute.internal <none> cirros-deployment-75c98888b9-lxq4k 1/1 Running 0 48m 10.47.255.250 ip-172-31-33-175.ap-northeast-1.compute.internal <none> cirros-deployment2-648b98685f-b8pxw 1/1 Running 0 13m 10.47.255.247 ip-172-31-25-102.ap-northeast-1.compute.internal <none> cirros-deployment2-648b98685f-nv7z9 1/1 Running 0 13m 10.47.255.248 ip-172-31-33-175.ap-northeast-1.compute.internal <none> cirros-deployment3-68fb484676-ct9q9 1/1 Running 0 18s 10.47.255.245 ip-172-31-25-102.ap-northeast-1.compute.internal <none> cirros-deployment3-68fb484676-mxbzq 1/1 Running 0 18s 10.47.255.246 ip-172-31-33-175.ap-northeast-1.compute.internal <none> [root@ip-172-31-19-25 ~]# |
-> 新容器依然可以被创建
1 2 3 4 5 6 7 8 9 10 11 |
[root@ip-172-31-25-102 ~]# contrail-status Pod Service Original Name State Id Status vrouter agent contrail-vrouter-agent running 9a46a1a721a7 Up 33 minutes vrouter nodemgr contrail-nodemgr running 11fb0a7bc86d Up 33 minutes vrouter kernel module is PRESENT == Contrail vrouter == nodemgr: active agent: active [root@ip-172-31-25-102 ~]# |
-> 具有新控制器的vRouter工作良好
1 2 3 4 5 6 7 8 9 |
/ # ping 10.47.255.250 PING 10.47.255.250 (10.47.255.250): 56 data bytes 64 bytes from 10.47.255.250: seq=0 ttl=63 time=1.781 ms 64 bytes from 10.47.255.250: seq=1 ttl=63 time=0.857 ms ^C --- 10.47.255.250 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.857/1.319/1.781 ms / # |
-> 在vRouter之间Ping成功了
向后兼容
由于有好几种更新集群的方法(就地、ISSU、是否为ifdown vhost0),因此方法的选择也是个重要的话题。
在讨论细节之前,让我先描述一下vrouter-agent up / down的行为,以及ifup vhost0 / ifdown vhost0的行为。
重新启动vrouter-agent时,一种假设是重新创建了vrouter-agent容器和vhost0。
实际上,事实并非如此,因为vhost0与vrouter.ko是紧密耦合的,需要从kernel中卸载vrouter.ko的同时将其删除。所以从操作角度来说,需要ifdown vhost0,那么不仅需要更新vrouter-agent,还需要更新vrouter.ko。(ifdown vhost0也将在内部执行rmmod vrouter)。
因此,要讨论向后兼容,需要研究下面三个主题。
1. controller与vrouter-agent的兼容性
• 如果没有向后兼容性,则需要ISSU
2. vrouter-agent与vrouter.ko的兼容性
• 如果没有向后兼容性,则需要ifdown vhost0,这将导致最少5-10秒的流量损失,因此实际上意味着需要将流量转移到其它节点,如实时迁移(live migration)
• 由于vrouter-agent使用netlink与vrouter.ko同步数据,因此架构更改可能导致vrouter-agent发生意外行为(例如Ksync logic上的vrouter-agent分段错误)
3. vrouter.ko与kernel的兼容性
• 如果没有向后兼容性,则需要更新kernel,因此这意味着需要将流量移至其它节点
• 当vrouter.ko具有不同的in-kernal API时,无法由kernel加载,并且无法创建vhost0和vrouter-agent
对于2和3,基于种种原因不可避免地需要进行kernel更新,因此一个可行的计划是,首先选择一个新的kernel版本,然后选择一个支持该kernel的vrouter-agent / vrouter.ko,再并检查当前使用的vrouter-agent是否可以与该版本的control一起使用。
• 如果运行良好,请使用就地更新;如果由于某些原因无法运行,或者需要回滚操作,此时需要使用ISSU
对于1,由于在导入config-api定义时ifmap会为每个版本维护white_list,
• void
IFMapGraphWalker::AddNodesToWhitelist():https://github.com/Juniper/contrail-controller/blob/master/src/ifmap/ifmap_graph_walker.cc#L349
根据我的尝试,它似乎具有不错的向后兼容性(由于路由信息更新与BGP类似,因此大多数情况下也应该可以正常工作)。
为了验证这一点,我尝试使用不同版本的模块进行该设置,看起来仍然可以正常工作。
I-1. config 2002-latest, control 2002-latest, vrouter 5.0-latest, openstack queens
I-2. config 2002-latest, control 5.0-latest, vrouter 5.0-latest, openstack queens
II-1. config 2002-latest, control 2002-latest, vrouter r5.1, kubernetes 1.12
注意:不幸的是,这种组合不能很好地工作(cni无法从vrouter-agent获取端口信息),我想这是由于5.0.x和5.1之间的cni版本更改(0.2.0-> 0.3.1)引起的。
II-2. config 2002-latest, control 2002-latest, vrouter 5.0-latest, kubernetes 1.12
因此,即使不需要立即更改kernel和vRouter版本,比较频繁地更新config / control也是一个好习惯,以修复可能的错误。