openstack(pike) dvr 中南北數據流向分析

1  在DVR環境中各組件的連接情況如圖


2 帶floatingip,從外部訪問虛機，ping floatingip 192.168.100.190
此時數據流如圖所示


2.1 獲取MAC
外網中的機器首先要通過 ARP 獲取虛機浮動 IP 對應的 MAC 地址。浮動 IP 并沒有配置在 fip 的端口上，因此 fip 無法直接響應 ARP 請求，那怎么辦呢？Neutron 在 fip NS 的 fg 端口上配置了 arp proxy，這樣，fip 既可以響應它自己的 interface 上的 IP 地址的 ARP 請求，也可以響應能通過它路由到的 IP 地址的 ARP 請求,
fip netns上配置的proxy_arp
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e sysctl net.ipv4.conf.fg-42d94dcb-8b.proxy_arp
net.ipv4.conf.fg-42d94dcb-8b.proxy_arp = 1
fip netns 收到 ARP 請求后，將其 fg interface 的 MAC 地址返回,外網中的機器獲取到虛機浮動 IP 的 MAC 地址后，發出 ICMP 網絡包
2.2 fip netns 路由到下qrouter netns
網絡包經過 br-ex，被 fip 的 fg 端口收到，查路由表，命中第一條路由，從其 fpr interface 發出，到達 169.254.106.114.
ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip route
169.254.106.114/31 dev fpr-b2ce2e13-1 proto kernel scope link src 169.254.106.115 
192.168.100.0/24 dev fg-42d94dcb-8b proto kernel scope link src 192.168.100.184 
192.168.100.190 via 169.254.106.114 dev fpr-b2ce2e13-1 
fpr interface 和rfp interface 是一堆veth pair , fpr interface 發出的包被 veth 另一端的 qrouter 的 rfp-b2ce2e13-1 interface 收到。
可以看到qrouter netns 的 rfp-b2ce2e13-1口地址是169.254.106.114/31


[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: rfp-b2ce2e13-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether b2:e7:86:35:3d:ac brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.106.114/31 scope global rfp-b2ce2e13-1
       valid_lft forever preferred_lft forever
    inet6 fe80::b0e7:86ff:fe35:3dac/64 scope link 
       valid_lft forever preferred_lft forever
134: qr-d2b8d53f-64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:02:97:ab brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global qr-d2b8d53f-64
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe02:97ab/64 scope link 
       valid_lft forever preferred_lft forever
2.3 DNAT
在 qrouter 上，首先做 DNAT
-A neutron-l3-agent-PREROUTING -d 192.168.100.190/32 -i rfp-b2ce2e13-1 -j DNAT --to-destination 172.17.0.7
DNAT后，dest ip 改為172.17.0.7
2.4 再次路由
查 qrouter 的 main 路由表，命中第二條，從qr-d2b8d53f-64 發出
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route
169.254.106.114/31 dev rfp-b2ce2e13-1 proto kernel scope link src 169.254.106.114 
172.17.0.0/16 dev qr-d2b8d53f-64 proto kernel scope link src 172.17.0.1 
數據包經qr-d2b8d53f-64,發到br-int 到達虛機


3 帶floatingip從虛機訪問外網，從172.17.0.7 ping 8.8.8.8
 3.1路由 
進入qroute netns,查看路由規則走route 16表
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local 
32766:from all lookup main 
32767:from all lookup default 
57483:from 172.17.0.7 lookup 16 
2886795265:from 172.17.0.1/16 lookup 2886795265 


[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table 16
default via 169.254.106.115 dev rfp-b2ce2e13-1 
3.2 SNAT
路由后會通過netfilter的POSTROUTING鏈中進行SNAT，將源IP改為192.168.100.190
[root@compute ~]# ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i rfp-b2ce2e13-1 ! -o rfp-b2ce2e13-1 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.100.190/32 -i rfp-b2ce2e13-1 -j DNAT --to-destination 172.17.0.7
-A neutron-l3-agent-float-snat -s 172.17.0.7/32 -j SNAT --to-source 192.168.100.190
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat


從rfp-b2ce2e13-1口將包發到169.254.106.115，rfp-b2ce2e13-1和fip netns的fpr-b2ce2e13-1，是一對veth pair，fpr-b2ce2e13-1收到數據包
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: fpr-b2ce2e13-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether aa:f7:5f:31:67:db brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.106.115/31 scope global fpr-b2ce2e13-1
       valid_lft forever preferred_lft forever
    inet6 fe80::a8f7:5fff:fe31:67db/64 scope link 
       valid_lft forever preferred_lft forever




135: fg-42d94dcb-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:c0:dc:94 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.184/24 brd 192.168.100.255 scope global fg-42d94dcb-8b
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec0:dc94/64 scope link 
       valid_lft forever preferred_lft forever
3.3 數據包在fip netns 中再次路由
包通過fpr-b2ce2e13-1發到169.254.106.115，后再在fip netns中查路由規則，路由
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip rule
0:from all lookup local 
32766:from all lookup main 
32767:from all lookup default 
2852022899:from all iif fpr-b2ce2e13-1 lookup 2852022899 
[root@compute ~]# ip netns exec fip-82d87e5f-167e-4525-90b2-c8f340630a1e ip route list table 2852022899
default via 192.168.100.1 dev fg-42d94dcb-8b 
從fg-42d94dcb-8b 口將包發到192.168.100.1，再有192.168.100.1繼續路由轉發


4 不帶浮動IP，虛機訪問外網
 4.1 計算節點qrouter 路由
查看路由規則
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local 
32766:from all lookup main 
32767:from all lookup default 
2886795265:from 172.17.0.1/16 lookup 2886795265 
查看路由表2886795265
ip netns exec qrouter-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table 2886795265
default via 172.17.0.10 dev qr-d2b8d53f-64
只有一條策略，經過qr口，發到下一個路由172.17.0.10
172.17.0.10　路由是在network節點的snat netns sg口
[root@network ~]# ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
93: sg-935eff5f-b8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:bf:6d:b5 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.10/16 brd 172.17.255.255 scope global sg-935eff5f-b8
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:febf:6db5/64 scope link 
       valid_lft forever preferred_lft forever
94: qg-2b2169ee-a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:9d:5c:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.187/24 brd 192.168.100.255 scope global qg-2b2169ee-a5
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe9d:5cbd/64 scope link 
       valid_lft forever preferred_lft forever
4.2 SNAT
network 節點的snat netns 做SNAT
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 iptables -t nat -S
-A neutron-l3-agent-snat -o qg-2b2169ee-a5 -j SNAT --to-source 192.168.100.187
4.3 snat netns路由
ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip rule
0:from all lookup local 
32766:from all lookup main 
32767:from all lookup default 
沒有額外策略，走main表
 ip netns exec snat-b2ce2e13-127d-491a-a669-ff07263e3751 ip route list table main
default via 192.168.100.1 dev qg-2b2169ee-a5 
172.17.0.0/16 dev sg-935eff5f-b8 proto kernel scope link src 172.17.0.10 
192.168.100.0/24 dev qg-2b2169ee-a5 proto kernel scope link src 192.168.100.187 
根據目的地址，命中第一條，包從qg-2b2169ee-a5口出去，再從192.168.100.1網關上繼續路由