容器网络其实并不难-伙伴云

网友投稿 1074 2022-05-30

使用容器总是感觉像使用魔法一样。对于那些理解底层原理的人来说容器很好用，但是对于不理解的人来说就是个噩梦。很幸运的是，我们已经研究容器技术很久了，甚至成功揭秘容器只是隔离并受限的Linux进程，运行容器并不需要镜像，以及另一个方面，构建镜像需要运行一些容器。

现在是时候解决容器网络问题了。或者更准确地说，单主机容器网络问题。本文会回答这些问题：

如何虚拟化网络资源，让容器认为自己拥有独占网络？

如何让容器们和平共处，之间不会互相干扰，并且能够互相通信？

从容器内部如何访问外部世界（比如，互联网）？

从外部世界如何访问某台机器上的容器呢（比如，端口发布）？

网络命名空间（namespace）

虚拟Ethernet设备（veth）

虚拟网络交换机（网桥）

IP路由和网络地址翻译（NAT）

$ vagrant init centos/8

$ vagrant up

$ vagrant ssh

[vagrant@localhost ~]$ uname -a

Linux localhost.localdomain 4.18.0-147.3.1.el8_1.x86_64

#!/usr/bin/env bash

echo "> Network devices"

ip link

echo -e "\n> Route table"

ip route

echo -e "\n> Iptables rules"

iptables --list-rules

$ sudo iptables -N ROOT_NS

$ sudo ./inspect-net-stack.sh

> Network devices

1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:e3:27:77 brd ff:ff:ff:ff:ff:ff

> Route table

default via 10.0.2.2 dev eth0 proto dhcp metric 100

10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100

> Iptables rules

-P INPUT ACCEPT

-P FORWARD ACCEPT

-P OUTPUT ACCEPT

-N ROOT_NS

$ sudo ip netns add netns0

$ ip netns

netns0

$ sudo nsenter --net=/var/run/netns/netns0 bash

# 新建的bash进程在netns0里

$ sudo ./inspect-net-stack.sh

> Network devices 1: lo: mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

> Route table

> Iptables rules

-P INPUT ACCEPT

-P FORWARD ACCEPT

-P OUTPUT ACCEPT

$ sudo ip link add veth0 type veth peer name ceth0

$ ip link

1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000

link/ether 52:54:00:e3:27:77 brd ff:ff:ff:ff:ff:ff

5: ceth0@veth0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 66:2d:24:e3:49:3f brd ff:ff:ff:ff:ff:ff

6: veth0@ceth0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 96:e8:de:1d:22:e0 brd ff:ff:ff:ff:ff:ff

$ sudo ip link set ceth0 netns netns0

# 列出所有设备，可以看到ceth0已经从root栈里消失了

$ ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000

link/ether 52:54:00:e3:27:77 brd ff:ff:ff:ff:ff:ff

6: veth0@if5: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 96:e8:de:1d:22:e0 brd ff:ff:ff:ff:ff:ff link-netns netns0

$ sudo ip link set veth0 up

$ sudo ip addr add 172.18.0.11/16 dev veth0

$ sudo nsenter --net=/var/run/netns/netns0

$ ip link set lo up

$ ip link set ceth0 up

$ ip addr add 172.18.0.10/16 dev ceth0

$ ip link

1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

5: ceth0@if6: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 66:2d:24:e3:49:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0

# 在netns0里ping root的 veth0

$ ping -c 2 172.18.0.11

PING 172.18.0.11 (172.18.0.11) 56(84) bytes of data.

64 bytes from 172.18.0.11: icmp_seq=1 ttl=64 time=0.038 ms

64 bytes from 172.18.0.11: icmp_seq=2 ttl=64 time=0.040 ms

--- 172.18.0.11 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 58ms

rtt min/avg/max/mdev = 0.038/0.039/0.040/0.001 ms

# 离开 netns0

$ exit

# 在root命名空间里ping ceth0

$ ping -c 2 172.18.0.10

PING 172.18.0.10 (172.18.0.10) 56(84) bytes of data.

64 bytes from 172.18.0.10: icmp_seq=1 ttl=64 time=0.073 ms

64 bytes from 172.18.0.10: icmp_seq=2 ttl=64 time=0.046 ms

--- 172.18.0.10 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 3ms

rtt min/avg/max/mdev = 0.046/0.059/0.073/0.015 ms

# 在 root 命名空间

$ ip addr show dev eth0

2: eth0: mtu 1500 qdisc fq_codel state UP group default qlen 1000

link/ether 52:54:00:e3:27:77 brd ff:ff:ff:ff:ff:ff

inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0

valid_lft 84057sec preferred_lft 84057sec

inet6 fe80::5054:ff:fee3:2777/64 scope link

valid_lft forever preferred_lft forever

# 记住这里IP是10.0.2.15

$ sudo nsenter --net=/var/run/netns/netns0

# 尝试ping主机的eth0

$ ping 10.0.2.15

connect: Network is unreachable

# 尝试连接外网

$ ping 8.8.8.8

connect: Network is unreachable

# 在netns0命名空间:

$ ip route

172.18.0.0/16 dev ceth0 proto kernel scope link src 172.18.0.10

# 从 root 命名空间

$ sudo ip netns add netns1

$ sudo ip link add veth1 type veth peer name ceth1

$ sudo ip link set ceth1 netns netns1

$ sudo ip link set veth1 up

$ sudo ip addr add 172.18.0.21/16 dev veth1

$ sudo nsenter --net=/var/run/netns/netns1

$ ip link set lo up

$ ip link set ceth1 up

$ ip addr add 172.18.0.20/16 dev ceth1

# 从netns1无法连通root 命名空间!

$ ping -c 2 172.18.0.21

PING 172.18.0.21 (172.18.0.21) 56(84) bytes of data.

From 172.18.0.20 icmp_seq=1 Destination Host Unreachable

From 172.18.0.20 icmp_seq=2 Destination Host Unreachable

--- 172.18.0.21 ping statistics ---

2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 55ms pipe 2

# 但是路由是存在的!

$ ip route

172.18.0.0/16 dev ceth1 proto kernel scope link src 172.18.0.20

# 离开 `netns1`

$ exit

# 从 root 命名空间无法连通`netns1`

$ ping -c 2 172.18.0.20

PING 172.18.0.20 (172.18.0.20) 56(84) bytes of data.

From 172.18.0.11 icmp_seq=1 Destination Host Unreachable

From 172.18.0.11 icmp_seq=2 Destination Host Unreachable

--- 172.18.0.20 ping statistics ---

2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 23ms pipe 2

# 从netns0可以连通`veth1`

$ sudo nsenter --net=/var/run/netns/netns0

$ ping -c 2 172.18.0.21

PING 172.18.0.21 (172.18.0.21) 56(84) bytes of data.

64 bytes from 172.18.0.21: icmp_seq=1 ttl=64 time=0.037 ms

64 bytes from 172.18.0.21: icmp_seq=2 ttl=64 time=0.046 ms

--- 172.18.0.21 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 33ms

rtt min/avg/max/mdev = 0.037/0.041/0.046/0.007 ms

# 但是仍然无法连通netns1

$ ping -c 2 172.18.0.20

PING 172.18.0.20 (172.18.0.20) 56(84) bytes of data.

From 172.18.0.10 icmp_seq=1 Destination Host Unreachable

From 172.18.0.10 icmp_seq=2 Destination Host Unreachable

--- 172.18.0.20 ping statistics ---

2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 63ms pipe 2

$ ip route

# ... 忽略无关行... #

172.18.0.0/16 dev veth0 proto kernel scope link src 172.18.0.11

172.18.0.0/16 dev veth1 proto kernel scope link src 172.18.0.21

$ sudo ip netns delete netns0

$ sudo ip netns delete netns1

$ sudo ip link delete veth0

$ sudo ip link delete ceth0

$ sudo ip link delete veth1

$ sudo ip link delete ceth1

$ sudo ip netns add netns0

$ sudo ip link add veth0 type veth peer name ceth0

$ sudo ip link set veth0 up

$ sudo ip link set ceth0 netns netns0

$ sudo nsenter --net=/var/run/netns/netns0

$ ip link set lo up

$ ip link set ceth0 up

$ ip addr add 172.18.0.10/16 dev ceth0

$ exit

$ sudo ip netns add netns1

$ sudo ip link add veth1 type veth peer name ceth1

$ sudo ip link set veth1 up

$ sudo ip link set ceth1 netns netns1

$ sudo nsenter --net=/var/run/netns/netns1

$ ip link set lo up

$ ip link set ceth1 up

$ ip addr add 172.18.0.20/16 dev ceth1

$ exit

$ ip route

default via 10.0.2.2 dev eth0 proto dhcp metric 100

10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100

$ sudo ip link add br0 type bridge

$ sudo ip link set br0 up

$ sudo ip link set veth0 master br0

$ sudo ip link set veth1 master br0

$ sudo nsenter --net=/var/run/netns/netns0

$ ping -c 2 172.18.0.20

PING 172.18.0.20 (172.18.0.20) 56(84) bytes of data.

64 bytes from 172.18.0.20: icmp_seq=1 ttl=64 time=0.259 ms

64 bytes from 172.18.0.20: icmp_seq=2 ttl=64 time=0.051 ms

--- 172.18.0.20 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 2ms

rtt min/avg/max/mdev = 0.051/0.155/0.259/0.104 ms

$ sudo nsenter --net=/var/run/netns/netns1

$ ping -c 2 172.18.0.10

PING 172.18.0.10 (172.18.0.10) 56(84) bytes of data.

64 bytes from 172.18.0.10: icmp_seq=1 ttl=64 time=0.037 ms

64 bytes from 172.18.0.10: icmp_seq=2 ttl=64 time=0.089 ms

--- 172.18.0.10 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 36ms

rtt min/avg/max/mdev = 0.037/0.063/0.089/0.026 ms

$ sudo nsenter --net=/var/run/netns/netns0

$ ip neigh

172.18.0.20 dev ceth0 lladdr 6e:9c:ae:02:60:de STALE

$ exit

$ sudo nsenter --net=/var/run/netns/netns1

$ ip neigh

172.18.0.10 dev ceth1 lladdr 66:f3:8c:75:09:29 STALE

$ exit

$ sudo nsenter --net=/var/run/netns/netns0

$ ping 10.0.2.15 # eth0 address

connect: Network is unreachable

$ ip route

172.18.0.0/16 dev ceth0 proto kernel scope link src 172.18.0.10

$ sudo ip addr add 172.18.0.1/16 dev br0

$ ip route

# ...忽略无关行 ...

172.18.0.0/16 dev br0 proto kernel scope link src 172.18.0.1

$ ping -c 2 172.18.0.10

PING 172.18.0.10 (172.18.0.10) 56(84) bytes of data.

64 bytes from 172.18.0.10: icmp_seq=1 ttl=64 time=0.036 ms

64 bytes from 172.18.0.10: icmp_seq=2 ttl=64 time=0.049 ms

--- 172.18.0.10 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 11ms

rtt min/avg/max/mdev = 0.036/0.042/0.049/0.009 ms

$ ping -c 2 172.18.0.20

PING 172.18.0.20 (172.18.0.20) 56(84) bytes of data.

64 bytes from 172.18.0.20: icmp_seq=1 ttl=64 time=0.059 ms

64 bytes from 172.18.0.20: icmp_seq=2 ttl=64 time=0.056 ms

--- 172.18.0.20 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 4ms

rtt min/avg/max/mdev = 0.056/0.057/0.059/0.007 ms

$ sudo nsenter --net=/var/run/netns/netns0

$ ip route add default via 172.18.0.1

$ ping -c 2 10.0.2.15

PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.

64 bytes from 10.0.2.15: icmp_seq=1 ttl=64 time=0.036 ms

64 bytes from 10.0.2.15: icmp_seq=2 ttl=64 time=0.053 ms

--- 10.0.2.15 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 14ms

rtt min/avg/max/mdev = 0.036/0.044/0.053/0.010 ms

# 为`netns1`也做上述配置

# 在 root 命名空间

sudo bash -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'

$ sudo nsenter --net=/var/run/netns/netns0

$ ping 8.8.8.8

# hung住了...

$ sudo iptables -t nat -A POSTROUTING -s 172.18.0.0/16 ! -o br0 -j MASQUERADE

sudo iptables -S

-P INPUT ACCEPT

-P FORWARD ACCEPT

-P OUTPUT ACCEPT

$ sudo iptables -t filter --list-rules

-P INPUT ACCEPT

-P FORWARD DROP

-P OUTPUT ACCEPT

-N DOCKER

-N DOCKER-ISOLATION-STAGE-1

-N DOCKER-ISOLATION-STAGE-2

-N DOCKER-USER

-A FORWARD -j DOCKER-USER

-A FORWARD -j DOCKER-ISOLATION-STAGE-1

-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

-A FORWARD -o docker0 -j DOCKER

-A FORWARD -i docker0 ! -o docker0 -j ACCEPT

-A FORWARD -i docker0 -o docker0 -j ACCEPT

-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5000 -j ACCEPT

-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2

-A DOCKER-ISOLATION-STAGE-1 -j RETURN

-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP

-A DOCKER-ISOLATION-STAGE-2 -j RETURN

-A DOCKER-USER -j RETURN

$ sudo iptables -t nat --list-rules

-P PREROUTING ACCEPT

-P INPUT ACCEPT

-P POSTROUTING ACCEPT

-P OUTPUT ACCEPT

-N DOCKER

-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER

-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 5000 -j MASQUERADE

-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER

-A DOCKER -i docker0 -j RETURN

-A DOCKER ! -i docker0 -p tcp -m tcp --dport 5005 -j DNAT --to-destination 172.17.0.2:5000

$ sudo iptables -t mangle --list-rules

-P PREROUTING ACCEPT

-P INPUT ACCEPT

-P FORWARD ACCEPT

-P OUTPUT ACCEPT

-P POSTROUTING ACCEPT

$ sudo iptables -t raw --list-rules

-P PREROUTING ACCEPT

-P OUTPUT ACCEPT

$ sudo nsenter --net=/var/run/netns/netns0

$ python3 -m http.server --bind 172.18.0.10 5000

# 从 root 命名空间

$ curl 172.18.0.10:5000

# ... 忽略无关行 ...

$ curl 10.0.2.15:5000

curl: (7) Failed to connect to 10.0.2.15 port 5000: Connection refused

# 外部流量

容器网络其实并不难

sudo iptables -t nat -A PREROUTING -d 10.0.2.15 -p tcp -m tcp --dport 5000 -j DNAT --to-destination 172.18.0.10:5000

# 本地流量 (因为它没有通过 PREROUTING chain)

sudo iptables -t nat -A OUTPUT -d 10.0.2.15 -p tcp -m tcp --dport 5000 -j DNAT --to-destination 172.18.0.10:5000

sudo modprobe br_netfilter

curl 10.0.2.15:5000

# ... 忽略无关行 ...

slirp4netns可以用完全非特权的方式将网络命名空间连接到Internet上，通过网络命名空间里的一个TAP设备连接到用户态的TCP/IP栈（slirp）。

https://docs.docker.com/network/#network-drivers

https://www.redhat.com/sysadmin/container-networking-podman

https://github.com/rootless-containers/slirp4netns

https://developers.redhat.com/blog/2018/10/22/introduction-to-linux-interfaces-for-virtual-networking/

容器网络虚拟化

为什么无法在电脑上、手机打开协作文档了？（文档在电脑上可以打开,但在手机里打不开）

1074 2022-05-30

容器 网络 其实 并不难

无法访问你试图使用的功能所在的网络位置。是什么原因？

为什么无法在电脑上、手机打开协作文档了？（文档在电脑上可以打开,但在手机里打不开）

图片边框里所有边框不见了怎么回事急急急（边框显示不出来）

推荐文章

企业生产管理是什么，企业生产管理软件

进盘点进销存软件排行榜前十名

进销存系统哪个简单好用？进销存系统优点

工厂生产管理（工厂生产管理流程及制度）

生产管理软件，机械制造业生产管理，制造业生产过程管理软件

进销存软件和ERP有什么区别？进销存与erp软件理解

进销存如何进行库存管理

如何利用excel制作销售订单管理系统？

数据库订单管理系统有哪些功能？数据库订单管理系统怎么设计？

什么是数据库管理系统？

最近发表

热评文章

零代码开发是什么？2022低代码平台排行榜

智能进销存库存管理系统（智慧进销存）

在线文档哪家强？8款在线文档编辑软件推荐

WPS2016怎么绘制简单的价格表?

什么是在线文档？怎么发在线文档

Excel项目进度表模板，简化您的项目进度管理

友情链接