Sonic测试研究（一）：测试架构介绍 | SDNLAB

作者简介：金利，盛科网络-SDN组测试工程师

Sonic项目简介

Sonic:Software for Open Networking in the Cloud
Sonic是基于Linux的开源网络操作系统，可以跑在多个不同芯片厂商交换机上
Sonic在2016年OCP峰会上，由微软发布
Sonic提供了全套的网络功能，并且都是开源的，如BGP、RDMA等协议
Sonic项目更多信息请查阅Github：github.com/Azure/SONic

Sonic测试框架介绍

Sonic项目专门成立了一个子项目Sonic-mgmt来负责Sonic交换机测试一块的事情

整个测试框架图如下

设备信息

这套测试框架由TestBed Server、两级Fanout交换机、Sonic被测交换机组成
TestBed Server上主要起两个docker（Sonic-mgmt、Docker-ptf）
- Sonic-mgmt 主要负责调配整个测试流程，包括启动协议VM、调用ptf等
- Docker-Ptf 主要负责收发报文以及数据平面的校验
Fanout交换机主要负责将Testbed Server上逻辑模拟出的端口跟被测设备物理对接起来
Sonic-DUT被测设备，启动Sonic软件的交换机

架构图逻辑解析

物理上TestBed-Server通过两级Fanout交换机连接到被测设备
- TestBed-Server内部通过Vlantag划分出多个逻辑端口(可以把每个逻辑端口想象成测试仪端口)
- RootFanout进行设备级的Vlan隔离，实现多组被测设备同时进行验证
- LeafFanout进行端口级的vlan隔离，实现Testbed逻辑端口与被测Sonic设备物理端口一一对应
逻辑上TestBed-Server上通过vlan隔离出来的各个逻辑端口与物理上被测Sonic设备进行一对一端口连接

TestBed Server内部结构

网络连接

TestBed Server有两个网络端口
- Mgmt口：管理端口，连接到实验室的管理网络
- Trunk口：数据端口，连接Fanout交换机，负责数据层面业务发包/收包验证

VM虚拟机

VM按照测试场景需要，由Mgmt-Sonic Docker负责调用启动，实际跑在TestBed Server上
- VM使用的是Arista vEOS 镜像
- VM主要负责跑各种协议，如BGP、LACP、LLDP，然后跟被测设备做协议层面的对接
- 每个VM由10个逻辑口，8个数据口逻辑上跟交换机各个物理端口相连，另外还有一个背板口负责VM之间互联，还有一个管理口

Docker-ptf

Ptf(python test framework) docker由Mgmt-Sonic Docker负责调用启动，跑在TestBed Server上
- Ptf docker可以按照实际场景，启动不同数量的逻辑端口与被测设备互联
- Ptf docker负责测试过程中数据层面，收发报文，以及报文验证
- Ptf 主要通过调用scapy库完成上述动作，并且可以灵活扩展，比如适配Ixia测试仪等

自动化脚本处理流程详解

控制执行层面
- 脚本以aniable-playbook的形式在Sonic-mgmt-docker上进行调用执行
- playbook中可以包含配置下发、状态查看、配置清空等动作
- playbook中可以通过调用ptf-docker来实现收发报文、验证报文
数据层面(以server发出为例)
- 数据层面Docker-Ptf、VM通过Server发送出来的报文都会携带不同的vlantag，用于区分内部逻辑端口（所以服务器数据出口是trunk口）
- RootFanout交换机收到带vlantag的报文之后，通过出口vlan过滤，将携带特定vlan范围的报文发送到特定的LeafFanout交换机上(所以Root Fanout入口和出口都是trunk口，只是过滤的范围不同)
- LeafFanout交换机收到带vlantag的报文之后，通过出口vlan过滤，并去除相应过滤tag之后，转发到被测设备的不同物理端口上，最终这个Tag对于被测设备来说是不可见的(所以LeafFanout连接RootFanout的是trunk口，连接被测设备的是Access端口)

简化逻辑图举例

这个示例图是去掉fanout交换机后的逻辑topo示例，摘自官网说明文档topo中其中一个
PTF这里的连接端口数是可以自己定义改变的
VM连接数量也是可以自己定义改变的

Sonic测试脚本

PTF发包原理

PTF发包实际都是调用的scapy来完成，跟oftest如出一辙
如果熟悉openflow oftest的小伙伴应该对这块会非常亲切


    &#39;&#39;&#39;
    When receiving gratuitous ARP packet, if it was not resolved in ARP table before,
    SONiC should discard the request and won&#39;t add ARP entry for the GARP
    &#39;&#39;&#39;
    def runTest(self):
        acs_mac = self.test_params[&#39;acs_mac&#39;]
        pkt = simple_arp_packet(pktlen=60,
                      eth_dst=&#39;ff:ff:ff:ff:ff:ff&#39;,
                      eth_src=&#39;00:05:07:08:09:0a&#39;,
                      vlan_vid=0,
                      vlan_pcp=0,
                      arp_op=1,
                      ip_snd=&#39;10.10.1.7&#39;,
                      ip_tgt=&#39;10.10.1.7&#39;,
                      hw_snd=&#39;00:05:07:08:09:0a&#39;,
                      hw_tgt=&#39;ff:ff:ff:ff:ff:ff&#39;,
                      )
        exp_pkt = simple_arp_packet(eth_dst=&#39;00:05:07:08:09:0a&#39;,
                        eth_src=acs_mac,
                        arp_op=2,
                        ip_snd=&#39;10.10.1.2&#39;,
                        ip_tgt=&#39;10.10.1.7&#39;,
                        hw_tgt=&#39;00:05:07:08:09:0a&#39;,
                        hw_snd=acs_mac,
                   )
        send_packet(self, self.test_params[&#39;port&#39;], pkt)
        verify_no_packet(self, exp_pkt, self.test_params[&#39;port&#39;])

'''

When receiving gratuitous ARP packet, if it was not resolved in ARP table before,

SONiC should discard the request and won't add ARP entry for the GARP

'''

def runTest(self):

acs_mac = self.test_params['acs_mac']

pkt = simple_arp_packet(pktlen=60,

eth_dst='ff:ff:ff:ff:ff:ff',

eth_src='00:05:07:08:09:0a',

vlan_vid=0,

vlan_pcp=0,

arp_op=1,

ip_snd='10.10.1.7',

ip_tgt='10.10.1.7',

hw_snd='00:05:07:08:09:0a',

hw_tgt='ff:ff:ff:ff:ff:ff',

)

exp_pkt = simple_arp_packet(eth_dst='00:05:07:08:09:0a',

eth_src=acs_mac,

arp_op=2,

ip_snd='10.10.1.2',

ip_tgt='10.10.1.7',

hw_tgt='00:05:07:08:09:0a',

hw_snd=acs_mac,

)

send_packet(self, self.test_params['port'], pkt)

verify_no_packet(self, exp_pkt, self.test_params['port'])

Sonic-Mgmt执行脚本

执行脚本、配置等操作流程实际都是调用的ansible-playbook来完成
脚本都比较长，下述截取部分arpall.yml中内容


    - name: wait for interfaces to be up after removed from portchannel
      pause: seconds=40
      when: (po1 is defined) or (po2 is defined)

    - name: copy test files over
      copy: src=roles/test/files/ptftests
            dest=/root
      delegate_to: &quot;{{ ptf_host }}&quot;

    - name: Clear DUT arp cache
      command: ip -stats neigh flush all
      ignore_errors: yes
      become: yes

    - name: Start PTF runner and Send correct unicast arp packets (10.10.1.3 to 10.10.1.2 with src_mac=00:06:07:08:09:00)
      include_tasks: ptf_runner.yml
      vars:
         ptf_test_name: ARP test
         ptf_test_dir: ptftests
         ptf_test_path: arptest.VerifyUnicastARPReply
         ptf_platform: remote
         ptf_platform_dir: ptftests
         ptf_test_params:
            - acs_mac=&#39;{{ ansible_interface_facts[intf1][&#39;macaddress&#39;] }}&#39;
            - port=&#39;{{ intf1_indice }}&#39;
         ptf_extra_options: &quot;--relax --debug info --log-file /tmp/arptest.VerifyUnicastARPReply.{{lookup(&#39;pipe&#39;,&#39;date +%Y-%m-%d-%H:%M:%S&#39;)}}.log &quot;

    - name: Get DUT arp table
      switch_arptable:

    - name: Check SONiC ARP table and confirm macaddress and interface are correct
      assert:
        that:
          - &quot;{{ arptable[&#39;v4&#39;][&#39;10.10.1.3&#39;][&#39;macaddress&#39;] == &#39;00:06:07:08:09:00&#39; }}&quot;
          - &quot;{{ arptable[&#39;v4&#39;][&#39;10.10.1.3&#39;][&#39;interface&#39;] == intf1 }}&quot;

    - name: Clear DUT arp cache
      command: ip -stats neigh flush all
      ignore_errors: yes
      become: yes

    # Send correct ARP request from correct interface, expecting normal behavior
    - name: PTF funner Send correct arp packets (10.10.1.3 to 10.10.1.2 with src_mac=00:06:07:08:09:0a)
      include_tasks: ptf_runner.yml
      vars:
         ptf_test_name: ARP test
         ptf_test_dir: ptftests
         ptf_test_path: arptest.ExpectReply
         ptf_platform: remote
         ptf_platform_dir: ptftests
         ptf_test_params:
            - acs_mac=&#39;{{ ansible_interface_facts[intf1][&#39;macaddress&#39;] }}&#39;
            - port=&#39;{{ intf1_indice }}&#39;
         ptf_extra_options: &quot;--relax --debug info --log-file /tmp/arptest.VerifyARPReply.{{lookup(&#39;pipe&#39;,&#39;date +%Y-%m-%d-%H:%M:%S&#39;)}}.log &quot;

    - name: Get DUT arp table
      switch_arptable:

    - name: Check SONiC ARP table and confirm macaddress and interface are correct
      assert:
        that:
          - &quot;{{ arptable[&#39;v4&#39;][&#39;10.10.1.3&#39;][&#39;macaddress&#39;] == &#39;00:06:07:08:09:0a&#39; }}&quot;
          - &quot;{{ arptable[&#39;v4&#39;][&#39;10.10.1.3&#39;][&#39;interface&#39;] == intf1 }}&quot;

    ## check DUT won&#39;t reply ARP and install ARP entry when ARP request coming from other interfaces
    - name: Clear DUT arp cache
      command: ip -stats neigh flush all
      ignore_errors: yes
      become: yes

- name: wait for interfaces to be up after removed from portchannel

pause: seconds=40

when: (po1 is defined) or (po2 is defined)

- name: copy test files over

copy: src=roles/test/files/ptftests

dest=/root

delegate_to: "{{ ptf_host }}"

- name: Clear DUT arp cache

command: ip -stats neigh flush all

ignore_errors: yes

become: yes

- name: Start PTF runner and Send correct unicast arp packets (10.10.1.3 to 10.10.1.2 with src_mac=00:06:07:08:09:00)

include_tasks: ptf_runner.yml

vars:

ptf_test_name: ARP test

ptf_test_dir: ptftests

ptf_test_path: arptest.VerifyUnicastARPReply

ptf_platform: remote

ptf_platform_dir: ptftests

ptf_test_params:

- acs_mac='{{ ansible_interface_facts[intf1]['macaddress'] }}'

- port='{{ intf1_indice }}'

ptf_extra_options: "--relax --debug info --log-file /tmp/arptest.VerifyUnicastARPReply.{{lookup('pipe','date +%Y-%m-%d-%H:%M:%S')}}.log "

- name: Get DUT arp table

switch_arptable:

- name: Check SONiC ARP table and confirm macaddress and interface are correct

assert:

that:

- "{{ arptable['v4']['10.10.1.3']['macaddress'] == '00:06:07:08:09:00' }}"

- "{{ arptable['v4']['10.10.1.3']['interface'] == intf1 }}"

- name: Clear DUT arp cache

command: ip -stats neigh flush all

ignore_errors: yes

become: yes

# Send correct ARP request from correct interface, expecting normal behavior

- name: PTF funner Send correct arp packets (10.10.1.3 to 10.10.1.2 with src_mac=00:06:07:08:09:0a)

include_tasks: ptf_runner.yml

vars:

ptf_test_name: ARP test

ptf_test_dir: ptftests

ptf_test_path: arptest.ExpectReply

ptf_platform: remote

ptf_platform_dir: ptftests

ptf_test_params:

- acs_mac='{{ ansible_interface_facts[intf1]['macaddress'] }}'

- port='{{ intf1_indice }}'

ptf_extra_options: "--relax --debug info --log-file /tmp/arptest.VerifyARPReply.{{lookup('pipe','date +%Y-%m-%d-%H:%M:%S')}}.log "

- name: Get DUT arp table

switch_arptable:

- name: Check SONiC ARP table and confirm macaddress and interface are correct

assert:

that:

- "{{ arptable['v4']['10.10.1.3']['macaddress'] == '00:06:07:08:09:0a' }}"

- "{{ arptable['v4']['10.10.1.3']['interface'] == intf1 }}"

## check DUT won't reply ARP and install ARP entry when ARP request coming from other interfaces

- name: Clear DUT arp cache

command: ip -stats neigh flush all

ignore_errors: yes

become: yes

最后号召一下，有没有也在搭这套测试环境的小伙伴加我微信feng_891005，一起讨论，一个人研究太累了。