openstack DVR openvswitch流表分析

openstack DVR环境组网

4node-twoselfnetwork
如上图:

  • 2计算节点 + 1 网络节点 +1控制节点
  • DVR配置计算节点无floating ip
  • 主要分析计算节点的ovs 流表策略
  • ubuntun 16.04 + openstack newton版本

计算节点compute的ovs流表

在此安装环境下计算节点共有3个ovs bridge(br-ex,br-int,br-tun)

  • br-ex:负责计算节点连接外网使用,需在dvr环境下配置floating ip vm数据流量才走此桥出外网
  • br-int:负责连接虚拟机所接入的linux bridge、router namespace
  • br-tun: 隧道所有的虚拟交换机,通过此隧道连接网络节点(dhcp、router、snat)、计算节点(虚拟机、router);完成大二层的需求

在对流表分析前,我们先了解网络拓扑,并获取接口和接口索引对应关系;

  • 查看compute br-int接口和接口索引映射关系

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    root@compute:~# ovs-ofctl show br-int
    OFPT_FEATURES_REPLY (xid=0x2): dpid:00005278ead35849
    n_tables:254, n_buffers:256
    capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
    actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
    1(patch-tun): addr:8e:c4:54:26:0f:46
    config: 0
    state: 0
    speed: 0 Mbps now, 0 Mbps max
    2(qvo06ec6270-33): addr:0a:7f:a0:b5:1f:b8
    config: 0
    state: 0
    current: 10GB-FD COPPER
    speed: 10000 Mbps now, 0 Mbps max
    3(qr-6474fe2d-84): addr:00:00:00:00:0e:00
    config: PORT_DOWN
    state: LINK_DOWN
    speed: 0 Mbps now, 0 Mbps max
    4(qvo2118dc24-92): addr:6e:1b:a5:df:d7:77
    config: 0
    state: 0
    current: 10GB-FD COPPER
    speed: 10000 Mbps now, 0 Mbps max
    5(int-br-ex): addr:aa:83:f4:ae:aa:41
    config: 0
    state: 0
    speed: 0 Mbps now, 0 Mbps max
    6(qr-d2e6fcc5-c7): addr:00:00:00:00:0e:00
    config: PORT_DOWN
    state: LINK_DOWN
    speed: 0 Mbps now, 0 Mbps max
    LOCAL(br-int): addr:52:78:ea:d3:58:49
    config: PORT_DOWN
    state: LINK_DOWN
    speed: 0 Mbps now, 0 Mbps max
    OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
    root@compute:~#
  • 查看compute br-tun接口和接口索引映射关系

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    root@compute:~# ovs-ofctl show br-tun
    OFPT_FEATURES_REPLY (xid=0x2): dpid:00005222d0a63e4f
    n_tables:254, n_buffers:256
    capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
    actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
    1(patch-int): addr:ae:2b:47:4b:15:6a
    config: 0
    state: 0
    speed: 0 Mbps now, 0 Mbps max
    2(vxlan-0a1e0aa0): addr:06:2d:8c:b2:d2:64
    config: 0
    state: 0
    speed: 0 Mbps now, 0 Mbps max
    3(vxlan-0a1e0a9a): addr:4e:7d:65:8d:09:09
    config: 0
    state: 0
    speed: 0 Mbps now, 0 Mbps max
    LOCAL(br-tun): addr:52:22:d0:a6:3e:4f
    config: PORT_DOWN
    state: LINK_DOWN
    speed: 0 Mbps now, 0 Mbps max
    OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
    root@compute:~#

接口对应关系已经标注在openvswitch DVR环境组网图上,见上图

分析 br-int流表

分析结果如图:

br-int-flow-table

步骤解析如下:

icmp6 协议交给table24处理

1
2
3
4
root@compute:~# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
cookie=0xbc602bf73a053ff8, duration=2944.223s, table=0, n_packets=0, n_bytes=0, idle_age=2944, priority=10,icmp6,in_port=2,icmp_type=136 actions=resubmit(,24)
cookie=0xbc602bf73a053ff8, duration=2938.510s, table=0, n_packets=0, n_bytes=0, idle_age=2938, priority=10,icmp6,in_port=4,icmp_type=136 actions=resubmit(,24)

2和4接口arp交给24表格处理

1
2
cookie=0xbc602bf73a053ff8, duration=2944.215s, table=0, n_packets=15, n_bytes=630, idle_age=547, priority=10,arp,in_port=2 actions=resubmit(,24)
cookie=0xbc602bf73a053ff8, duration=2938.505s, table=0, n_packets=10, n_bytes=420, idle_age=6, priority=10,arp,in_port=4 actions=resubmit(,24)

5接口流入的数据包全部丢弃

1
cookie=0xbc602bf73a053ff8, duration=2968.234s, table=0, n_packets=4699, n_bytes=1348102, idle_age=1, priority=2,in_port=5 actions=drop

2和4口其它协议数据包统统交给25表格处理

1
2
cookie=0xbc602bf73a053ff8, duration=2944.232s, table=0, n_packets=927, n_bytes=106878, idle_age=1, priority=9,in_port=2 actions=resubmit(,25)
cookie=0xbc602bf73a053ff8, duration=2938.520s, table=0, n_packets=1022, n_bytes=126940, idle_age=1, priority=9,in_port=4 actions=resubmit(,25)

其它数据包像正常交换机数据一样转发数据

1
cookie=0xbc602bf73a053ff8, duration=2968.576s, table=0, n_packets=2332, n_bytes=240272, idle_age=1, priority=0 actions=NORMAL

23表格默认全部丢弃数据包

1
cookie=0xbc602bf73a053ff8, duration=2968.580s, table=23, n_packets=0, n_bytes=0, idle_age=2968, priority=0 actions=drop

表24 处理ipv6 nd

1
2
cookie=0xbc602bf73a053ff8, duration=2944.226s, table=24, n_packets=0, n_bytes=0, idle_age=2944, priority=2,icmp6,in_port=2,icmp_type=136,nd_target=fe80::f816:3eff:fef3:31f5 actions=NORMAL
cookie=0xbc602bf73a053ff8, duration=2938.513s, table=24, n_packets=0, n_bytes=0, idle_age=2938, priority=2,icmp6,in_port=4,icmp_type=136,nd_target=fe80::f816:3eff:fe3c:d256 actions=NORMAL

表24 接口2或4 收到 172.16.1.3 1.1.1.11 的arp信息交给表25处理

1
2
cookie=0xbc602bf73a053ff8, duration=2944.219s, table=24, n_packets=15, n_bytes=630, idle_age=547, priority=2,arp,in_port=2,arp_spa=172.16.1.3 actions=resubmit(,25)
cookie=0xbc602bf73a053ff8, duration=2938.507s, table=24, n_packets=10, n_bytes=420, idle_age=6, priority=2,arp,in_port=4,arp_spa=1.1.1.11 actions=resubmit(,25)

表24 如果上述未处理完成,将剩余数据包丢去

1
cookie=0xbc602bf73a053ff8, duration=2968.573s, table=24, n_packets=0, n_bytes=0, idle_age=2968, priority=0 actions=drop

表25 接口2或则4 源mac是fa:16:3e:f3:31:f5 fa:16:3e:3c:d2:56 的来的数据包正常处理

1
2
cookie=0xbc602bf73a053ff8, duration=2944.242s, table=25, n_packets=926, n_bytes=106092, idle_age=1, priority=2,in_port=2,dl_src=fa:16:3e:f3:31:f5 actions=NORMAL
cookie=0xbc602bf73a053ff8, duration=2938.524s, table=25, n_packets=1016, n_bytes=125944, idle_age=1, priority=2,in_port=4,dl_src=fa:16:3e:3c:d2:56 actions=NORMAL

分析 br-tun流表

分析结果如图:

br-tun-flowtabble

步骤解析如下:

内网接口 交给table2处理

1
2
3
root@compute:~# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
cookie=0xb9ef2c735af9d49c, duration=2988.059s, table=0, n_packets=299, n_bytes=30921, idle_age=208, priority=1,in_port=1 actions=resubmit(,2)

外网接口交给table4处理

1
2
cookie=0xb9ef2c735af9d49c, duration=2963.688s, table=0, n_packets=154, n_bytes=12924, idle_age=208, priority=1,in_port=2 actions=resubmit(,4)
cookie=0xb9ef2c735af9d49c, duration=2963.630s, table=0, n_packets=200, n_bytes=19684, idle_age=210, priority=1,in_port=3 actions=resubmit(,4)

其它:

1
cookie=0xb9ef2c735af9d49c, duration=2988.034s, table=0, n_packets=0, n_bytes=0, idle_age=2988, priority=0 actions=drop

单播交给表20处理

1
cookie=0xb9ef2c735af9d49c, duration=2988.027s, table=2, n_packets=209, n_bytes=20555, idle_age=208, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)

多播交给表22处理

1
cookie=0xb9ef2c735af9d49c, duration=2988.009s, table=2, n_packets=90, n_bytes=10366, idle_age=210, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)

其它:

1
cookie=0xb9ef2c735af9d49c, duration=2988.006s, table=3, n_packets=0, n_bytes=0, idle_age=2988, priority=0 actions=drop

从外网过来的数据包,去除tunid添加对应的vlanid后将数据包交给table10处理

1
2
cookie=0xb9ef2c735af9d49c, duration=2975.899s, table=4, n_packets=206, n_bytes=19127, idle_age=208, priority=1,tun_id=0x44 actions=mod_vlan_vid:1,resubmit(,10)
cookie=0xb9ef2c735af9d49c, duration=2963.083s, table=4, n_packets=148, n_bytes=13481, idle_age=208, priority=1,tun_id=0x5b actions=mod_vlan_vid:2,resubmit(,10)

其它:

1
2
cookie=0xb9ef2c735af9d49c, duration=2988.005s, table=4, n_packets=0, n_bytes=0, idle_age=2988, priority=0 actions=drop
cookie=0xb9ef2c735af9d49c, duration=2987.998s, table=6, n_packets=0, n_bytes=0, idle_age=2987, priority=0 actions=drop

学习mac等相关信息,将信息放到表20,将数据包从pach-int发出交给内网处理

1
cookie=0xb9ef2c735af9d49c, duration=2987.994s, table=10, n_packets=354, n_bytes=32608, idle_age=208, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xb9ef2c735af9d49c,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:1

发往目的mac为fa:16:3e:3e:69:70 tag为1 (ip为172.16.1.9虚拟机数据包)被去除vlan 加上tunid,交给compute1 连接的隧道接口

1
cookie=0xb9ef2c735af9d49c, duration=2949.737s, table=20, n_packets=66, n_bytes=6977, idle_age=2871, priority=2,dl_vlan=1,dl_dst=fa:16:3e:3e:69:70 actions=strip_vlan,load:0x44->NXM_NX_TUN_ID[],output:3

发给172.16.1.2 dhcp 服务namespace数据包去除vlan后通过隧道交给网络节点

1
cookie=0xb9ef2c735af9d49c, duration=2937.625s, table=20, n_packets=1, n_bytes=42, idle_age=2886, priority=2,dl_vlan=1,dl_dst=fa:16:3e:cb:c0:9e actions=strip_vlan,load:0x44->NXM_NX_TUN_ID[],output:2

发给1.1.1.2 dhcp 服务namespace数据包去除vlan后通过隧道交给网络节点

1
cookie=0xb9ef2c735af9d49c, duration=2937.136s, table=20, n_packets=1, n_bytes=42, idle_age=2879, priority=2,dl_vlan=2,dl_dst=fa:16:3e:b0:bd:78 actions=strip_vlan,load:0x5b->NXM_NX_TUN_ID[],output:2

发给1.1.1.12的snat中网关数据包去除vlan后通过隧道交给网络节点

1
cookie=0xb9ef2c735af9d49c, duration=2926.229s, table=20, n_packets=0, n_bytes=0, idle_age=2958, priority=2,dl_vlan=2,dl_dst=fa:16:3e:6c:ab:eb actions=strip_vlan,load:0x5b->NXM_NX_TUN_ID[],output:2

发往172.16.1.4的snat中网关数据包去除vlan后通过隧道交给网络节点

1
cookie=0xb9ef2c735af9d49c, duration=2898.976s, table=20, n_packets=1, n_bytes=42, idle_age=2889, priority=2,dl_vlan=1,dl_dst=fa:16:3e:6b:31:95 actions=strip_vlan,load:0x44->NXM_NX_TUN_ID[],output:2
  • 学习到的表项

    1
    2
    cookie=0xb9ef2c735af9d49c, duration=210.599s, table=20, n_packets=2, n_bytes=196, hard_timeout=300, idle_age=209, hard_age=208, priority=1,vlan_tci=0x0002/0x0fff,dl_dst=fa:16:3e:1f:95:2c actions=load:0->NXM_OF_VLAN_TCI[],load:0x5b->NXM_NX_TUN_ID[],output:2
    cookie=0xb9ef2c735af9d49c, duration=210.597s, table=20, n_packets=2, n_bytes=264, hard_timeout=300, idle_age=208, hard_age=208, priority=1,vlan_tci=0x0001/0x0fff,dl_dst=fa:16:3e:2c:c3:0d actions=load:0->NXM_OF_VLAN_TCI[],load:0x44->NXM_NX_TUN_ID[],output:2
  • 未命中表20其它策略,统统交给流表22处理

    1
    cookie=0xb9ef2c735af9d49c, duration=2987.986s, table=20, n_packets=5, n_bytes=418, idle_age=210, priority=0 actions=resubmit(,22)

表22去除vlan 附加上tunid将数据包从隧道发出

1
2
cookie=0xb9ef2c735af9d49c, duration=2926.237s, table=22, n_packets=18, n_bytes=2034, idle_age=210, priority=1,dl_vlan=2 actions=strip_vlan,load:0x5b->NXM_NX_TUN_ID[],output:2,output:3
cookie=0xb9ef2c735af9d49c, duration=2898.979s, table=22, n_packets=23, n_bytes=2374, idle_age=568, priority=1,dl_vlan=1 actions=strip_vlan,load:0x44->NXM_NX_TUN_ID[],output:2,output:3

表22其它:

1
cookie=0xb9ef2c735af9d49c, duration=2987.979s, table=22, n_packets=54, n_bytes=6376, idle_age=2959, priority=0 actions=drop

分析br-ex流表

1
2
3
4
root@compute:~# ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
cookie=0x92b626c066ab8e9d, duration=13306.841s, table=0, n_packets=224, n_bytes=20652, idle_age=1596, priority=2,in_port=2 actions=drop
cookie=0x92b626c066ab8e9d, duration=13306.862s, table=0, n_packets=277292, n_bytes=89976779, idle_age=0, priority=0 actions=NORMAL

如上流表:默认2三层转发策略,且丢弃从外网收到的数据包