Configuring static routing
In the guide below, we will demonstrate static routing using a virtual networking lab using vmm.
In order to save on costs, we will be using the reserved IP address prefix 10/8 and then translate the range to a public IP address. Here is the topology:
R1 <-----> Internet
(10.1/16)
10.2.1.1
^
|
|
veb12
|
|
V
10.2.2.1
R2 10.3.2.1 <-veb23-> 10.3.2.2 R3 10.5.3.1 <--veb35--> 10.5.5.1 R5
(10.2/16) (10.3/16) (10.5/16)
10.4.2.1 10.3.4.2
^ ^
\ /
veb24 /
\ veb34
\ /
------> R4 <-----
10.4.4.1 (10.4/16) 10.3.4.1
Each R# in the above topology represents a router, which we simulate with a virtual machine with multiple interfaces. We will use veb(4) to link two routers together. Because there are multiple interfaces on each router, no three nodes will ever share the same broadcast domain. This ensures that routing will be required to traverse the network.
In the hypervisor, we configure the proper interfaces:
host# cat /etc/vm.conf
socket owner :vmdusers
switch "switch12" {
locked lladdr
interface veb12
}
switch "switch23" {
locked lladdr
interface veb23
}
switch "switch24" {
locked lladdr
interface veb24
}
switch "switch34" {
locked lladdr
interface veb34
}
switch "switch35" {
locked lladdr
interface veb35
}
bsdiso="/home/iso/install75.iso"
vm "r2" {
owner $USER
memory 1G
cdrom $bsdiso
disk /home/$USER/r2.qcow2 format qcow2
interface tap22 {
locked lladdr e8:8b:22:22:22:22
switch "switch12"
}
interface tap32 {
locked lladdr e8:8b:32:32:32:32
switch "switch23"
}
interface tap42 {
locked lladdr e8:8b:42:42:42:42
switch "switch24"
}
}
vm "r3" {
owner $USER
memory 1G
cdrom $bsdiso
disk /home/$USER/r3.qcow2 format qcow2
interface tap322 {
locked lladdr e8:8b:32:02:32:02
switch "switch23"
}
interface tap342 {
locked lladdr e8:8b:34:02:34:02
switch "switch34"
}
interface tap53 {
locked lladdr e8:8b:53:53:53:53
switch "switch35"
}
}
vm "r4" {
owner $USER
memory 1G
cdrom $bsdiso
disk /home/$USER/r4.qcow2 format qcow2
interface tap44 {
switch "switch24"
locked lladdr e8:8b:44:44:44:44
}
interface tap34 {
locked lladdr e8:8b:34:34:34:34
switch "switch34"
}
}
vm "r5" {
owner $USER
memory 1G
cdrom $bsdiso
disk /home/$USER/r5.qcow2 format qcow2
interface tap55 {
switch "switch35"
locked lladdr e8:8b:55:55:55:55
}
}
NOTE: You will need to create necessary tap devices:
# cd /dev
# sh MAKEDEV tap{22,32,42,322,342,53,44,34,55}
We also create the qcow2 images and install the system:
$ vmctl create -s 20G $HOME/r2.qcow2 $ vmctl create -s 20G $HOME/r3.qcow2 $ vmctl create -s 20G $HOME/r4.qcow2 $ vmctl create -s 20G $HOME/r5.qcow2
In /etc/pf.conf?, we add a rule similar to the following for performing NAT:
match out on egress from !(egress:network) to any nat-to (egress:0)
Now all IPs from 10/8 will be NAT'd to our host's public IP address.
Then reload the ruleset:
host# pfctl -f /etc/pf.conf
We configure the proper interfaces on the host and sysctl.conf(5):
host# cat /etc/hostname.veb12 add vport21 up host# cat /etc/hostname.veb23 up host# cat /etc/hostname.veb24 up host# cat /etc/hostname.veb34 up host# cat /etc/hostname.veb35 up host# cat /etc/hostname.vport11 inet 10.1.1.1 0xffff0000 up host# cat /etc/hostname.vport21 inet 10.2.1.1 0xffff0000 !route add -inet 10/8 10.2.2.1 up host# cat /etc/sysctl.conf net.inet.ip.arpq.maxlen=1024 net.inet.ip.forwarding=1 net.inet6.ip6.forwarding=1
Next, we configure the virtual machines:
r2# cat /etc/hostname.vio0 inet 10.2.2.1 0xffff0000 !route add -inet 10.1/16 10.2.1.1 !route add -inet default 10.2.1.1 up r2# cat /etc/hostname.vio1 inet 10.3.2.1 0xffffff00 !route add -inet 10.3/16 10.3.2.2 !route add -inet 10.5/16 10.3.2.2 up r2# cat /etc/hostname.vio2 inet 10.4.2.1 0xffff0000 !route add -inet 10.4/16 10.4.4.1 up r2# cat /etc/sysctl.conf net.inet.ip.forwarding=1 r3# cat /etc/hostname.vio0 inet 10.3.2.2 0xffffff00 !route add -inet 10.2/16 10.3.2.1 !route add -inet default 10.3.2.1 up r3# cat /etc/hostname.vio1 inet 10.3.4.2 0xffffff00 !route add -inet 10.4/16 10.3.4.1 up r3# cat /etc/hostname.vio2 inet 10.5.3.1 0xffff0000 !route add -inet 10.5/16 10.5.5.1 up r3# cat /etc/sysctl.conf net.inet.ip.forwarding=1 r4# cat /etc/hostname.vio0 inet 10.4.4.1 0xffff0000 !route add -inet 10.2/16 10.4.2.1 !route add -inet default 10.4.2.1 up r4# cat /etc/hostname.vio1 inet 10.3.4.1 0xffffff00 !route add -inet 10.3/16 10.3.4.2 !route add -inet 10.5/16 10.3.4.2 up r4# cat /etc/sysctl.conf net.inet.ip.forwarding=1 r5# cat /etc/hostname.vio0 inet 10.5.5.1 0xffff0000 !route add -inet 10.3/16 10.5.3.1 !route add -inet default 10.5.3.1 up r5# cat /etc/sysctl.conf net.inet.ip.forwarding=1
Note: Be careful to input the correct subnet masks. Interfaces connecting to 10.3/16 have a longer mask.
In this basic setup, we add static routes to allow packets to be forwarded inside the network.
Notice the routes that are chosen. If R5 runs:
r5# ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1): 56 data bytes 64 bytes from 1.1.1.1: icmp_seq=0 ttl=53 time=4.181 ms 64 bytes from 1.1.1.1: icmp_seq=1 ttl=53 time=4.353 ms
You can run tcpdump on each of the routers' interfaces to see which interfaces receive the packet:
# tcpdump -ne -i if0 'icmp and host 1.1.1.1'
Replace if0 with vio0, vio1, or vio2. You can also run:
To get from R5 to the internet, R5 passes through R3, R2, and R1, but does not pass through R4:
r3# tcpdump -ne -i vio0 'icmp and host 1.1.1.1' tcpdump: listening on vio0, link-type EN10MB 08:47:30.937682 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 08:47:30.941063 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] 08:47:31.937602 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 08:47:31.941035 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] r4# tcpdump -ne -i vio0 'icmp and host 1.1.1.1' tcpdump: listening on vio0, link-type EN10MB ^C 0 packets received by filter 0 packets dropped by kernel
The route command can also tell you where packets will be sent:
r5# route -n get 1.1.1.1
route to: 1.1.1.1
destination: 0.0.0.0
mask: 0.0.0.0
gateway: 10.5.3.1
interface: vio0
if address: 10.5.5.1
priority: 8 (static)
flags: <UP,GATEWAY,DONE,STATIC>
use mtu expire
34 0 0
Route priorities
By changing the routing tables, we can change the path that packets take.
First, notice how R3 can take two paths to reach R2. It can reach R2 directly, or it can first send packets to R4 which then forwards them to R2.
Let's have R3 send packets to R4 in order to reach R2. However, let's have R2 send packets to R3 directly. This will create asymmetric routing.
r2# cat /etc/hostname.vio0 inet 10.2.2.1 0xffff0000 !route add -inet 10.1/16 10.2.1.1 !route add -inet default 10.2.1.1 up r2# cat /etc/hostname.vio1 inet 10.3.2.1 0xffffff00 !route add -inet -priority 2 -mpath 10.3/16 10.3.2.2 !route add -inet -priority 2 -mpath 10.5/16 10.3.2.2 up r2# cat /etc/hostname.vio2 inet 10.4.2.1 0xffff0000 !route add -inet -mpath 10.3/16 10.4.4.1 !route add -inet 10.4/16 10.4.4.1 !route add -inet -mpath 10.5/16 10.4.4.1 up r2# cat /etc/sysctl.conf net.inet.ip.forwarding=1 net.inet.ip.multipath=1 r3# cat /etc/hostname.vio0 inet 10.3.2.2 0xffffff00 !route add -inet -mpath 10.2/16 10.3.2.1 !route add -inet -mpath default 10.3.2.1 up r3# cat /etc/hostname.vio1 inet 10.3.4.2 0xffffff00 !route add -inet -priority 2 -mpath 10.4/16 10.3.4.1 !route add -inet -priority 2 -mpath default 10.3.4.1 up r3# cat /etc/hostname.vio2 inet 10.5.3.1 0xffff0000 !route add -inet 10.5/16 10.5.5.1 up r3# cat /etc/sysctl.conf net.inet.ip.forwarding=1 net.inet.ip.multipath=1 r4# cat /etc/hostname.vio0 inet 10.4.4.1 0xffff0000 !route add -inet 10.1/16 10.4.2.1 !route add -inet -priority 2 -mpath 10.2/16 10.4.2.1 !route add -inet -priority 2 -mpath default 10.4.2.1 up r4# cat /etc/hostname.vio1 inet 10.3.4.1 0xffffff00 !route add -inet 10.3/16 10.3.4.2 !route add -inet 10.5/16 10.3.4.2 !route add -inet -mpath default 10.3.4.2 up r4# cat /etc/sysctl.conf net.inet.ip.forwarding=1 net.inet.ip.multipath=1 r5# cat /etc/hostname.vio0 inet 10.5.5.1 0xffff0000 !route add -inet 10.3/16 10.5.3.1 !route add -inet default 10.5.3.1 up r5# cat /etc/sysctl.conf net.inet.ip.forwarding=1 net.inet.ip.multipath=1
Note: we must add the net.inet.ip.multipath sysctl in order to handle multipath routing.
Note: route priority 1 is reserved and cannot be used, so we use the next lowest priority, 2.
From route(8):
From ifconfig(8):
On R5, try running:
r5# ping 1.1.1.1
You can see ping requests coming from R3 (10.4.2.1) to R4 vio0 (10.4.4.1), but no ping replies; and you can see ping replies coming from R2 (10.3.2.1) to R3 (10.3.2.2), but no ping requests:
r3# tcpdump -ne -i vio0 'icmp and host 1.1.1.1' tcpdump: listening on vio0, link-type EN10MB 21:08:40.272715 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] 21:08:41.272550 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] 21:08:42.272799 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] r4# tcpdump -ne -i vio0 'icmp and host 1.1.1.1' tcpdump: listening on vio0, link-type EN10MB 21:08:44.270061 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 21:08:45.270192 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 21:08:46.270045 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request
Let's fix up the routes so that we have symmetric routing and multiple paths defined. The routers will choose routes based on their priorities.
r2# cat /etc/hostname.vio0 inet 10.2.2.1 0xffff0000 !route add -inet 10.1/16 10.2.1.1 !route add -inet default 10.2.1.1 up r2# cat /etc/hostname.vio1 inet 10.3.2.1 0xffffff00 !route add -inet -priority 2 -mpath 10.3/16 10.3.2.2 !route add -inet -priority 2 -mpath 10.5/16 10.3.2.2 up r2# cat /etc/hostname.vio2 inet 10.4.2.1 0xffff0000 !route add -inet -mpath 10.3/16 10.4.4.1 !route add -inet 10.4/16 10.4.4.1 !route add -inet -mpath 10.5/16 10.4.4.1 up r2# cat /etc/sysctl.conf net.inet.ip.forwarding=1 net.inet.ip.multipath=1 r3# cat /etc/hostname.vio0 inet 10.3.2.2 0xffffff00 !route add -inet -mpath -priority 2 10.1/16 10.2.3.1 !route add -inet -mpath -priority 2 10.2/16 10.3.2.1 !route add -inet -mpath -priority 2 default 10.3.2.1 up r3# cat /etc/hostname.vio1 inet 10.3.4.2 0xffffff00 !route add -inet -mpath 10.1/16 10.3.4.1 !route add -inet -mpath 10.2/16 10.3.4.1 !route add -inet 10.4/16 10.3.4.1 !route add -inet -mpath default 10.3.4.1 up r3# cat /etc/hostname.vio2 inet 10.5.3.1 0xffff0000 !route add -inet 10.5/16 10.5.5.1 up r3# cat /etc/sysctl.conf net.inet.ip.forwarding=1 net.inet.ip.multipath=1 r4# cat /etc/hostname.vio0 inet 10.4.4.1 0xffff0000 !route add -inet 10.1/16 10.4.2.1 !route add -inet 10.2/16 10.4.2.1 !route add -inet default 10.4.2.1 up r4# cat /etc/hostname.vio1 inet 10.3.4.1 0xffffff00 !route add -inet 10.3/16 10.3.4.2 !route add -inet 10.5/16 10.3.4.2 up r4# cat /etc/sysctl.conf net.inet.ip.forwarding=1 net.inet.ip.multipath=1 r5# cat /etc/hostname.vio0 inet 10.5.5.1 0xffff0000 !route add -inet 10.3/16 10.5.3.1 !route add -inet default 10.5.3.1 up r5# cat /etc/sysctl.conf net.inet.ip.forwarding=1 net.inet.ip.multipath=1
Now, on R5, run ping:
r5# ping 1.1.1.1
Run tcpdump to notice that no packets pass through R4, and all packets pass through R3, as expected:
r3# tcpdump -ne -i vio0 'icmp and host 1.1.1.1' tcpdump: listening on vio0, link-type EN10MB 22:52:10.308964 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 22:52:10.312342 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] 22:52:11.308979 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 22:52:11.312279 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] 22:52:12.308964 e8:8b:32:02:32:02 e8:8b:32:32:32:32 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 22:52:12.312357 e8:8b:32:32:32:32 e8:8b:32:02:32:02 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] r4# tcpdump -ne -i vio0 'icmp and host 1.1.1.1' tcpdump: listening on vio0, link-type EN10MB ^C 0 packets received by filter 0 packets dropped by kernel
Now, let's see what happens when one of the links breaks. We will bring down the vio0 interface on R3, to break the link between R2 and R3:
r3# ifconfig vio0 down
On R2, we change route priorities:
r2# route change -inet -priority 10 10.5/16 10.3.2.1 change net 10.5/16: gateway 10.3.2.1 r2# route change -inet -priority 2 10.5/16 10.4.4.1 change net 10.5/16: gateway 10.4.4.1
On R5, ping should still work:
r5# ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1): 56 data bytes 64 bytes from 1.1.1.1: icmp_seq=0 ttl=52 time=4.831 ms 64 bytes from 1.1.1.1: icmp_seq=1 ttl=52 time=4.349 ms
However, it now takes a different route. Notice that R4 now receives all packets:
r4# tcpdump -ne -i vio0 'icmp and host 1.1.1.1' tcpdump: listening on vio0, link-type EN10MB 00:25:59.811394 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 00:25:59.814762 e8:8b:42:42:42:42 e8:8b:44:44:44:44 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] 00:26:00.811231 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 00:26:00.814731 e8:8b:42:42:42:42 e8:8b:44:44:44:44 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28] 00:26:01.811288 e8:8b:44:44:44:44 e8:8b:42:42:42:42 0800 98: 10.5.5.1 > 1.1.1.1: icmp: echo request 00:26:01.814664 e8:8b:42:42:42:42 e8:8b:44:44:44:44 0800 98: 1.1.1.1 > 10.5.5.1: icmp: echo reply [tos 0x28]