Site to site with OpenVPN with TLS and FirewallD on Debian
So I needed to setup a site to site transport between our two datacenters, so our internal network will be able to communicate between the two datacenters easily.
There are a lot of ways to setup site to site VPN link between two networks, in our case let’s take a look at one way using OpenVPN.
This is quite long tutorial so get yourself a bit comfortable so we can get started.
Our network looks like this
Datacenter at location A has internal network 10.5.200.20 on the interface eth1 and 94.x.x.x.x on eth0, we will configure the openvpn server on this location. Datacenter at location B has internal network 10.0.0.5 on the interface eth1 and 93.x.x.x.x on eth0.
Both machine have two network interfaces eth0 which is connected to the internet, and eth1 which is connected the the LAN
The IP subnet has to be different for this to work, if they are on the same subnet this won’t work.
VPN gateway at location A
So to begin login to the machine in the location A
Forwarding
$ cat <<EOF > /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
$ sysctl -p /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
Installation
apt-get install openvpn easy-rsa firewalld
Firewall
Let’s configure the firewall to allow us to connect to the machine and all the necessary configuration needed for it
$ firewall-cmd --permanent --zone=public --add-source=93.x.x.x/32
$ firewall-cmd --permanent --zone=trusted --add-interface=eth1
$ firewall-cmd --permanent --zone=trusted --add-interface=tun0
$ firewall-cmd --permanent --zone=trusted --add-masquerade
$ DEV=$(ip route get 8.8.8.8 | awk 'NR==1 {print $(NF-2)}')
$ firewall-cmd --permanent --direct --passthrough ipv4 -t nat -A POSTROUTING -s 172.16.1.0/24 -o $DEV -j MASQUERADE
$ firewall-cmd --reload
Generating OpenVPN keys
After we have installed all this we need to generate some configuration and tls keys for us to use.
$ make-cadir ~/ca
$ cd ~/ca
$ ls -lah
drwx------ 2 root root 4.0K Feb 5 21:23 .
drwxr-xr-x 4 root root 4.0K Feb 5 21:23 ..
lrwxrwxrwx 1 root root 28 Feb 5 21:23 build-ca -> /usr/share/easy-rsa/build-ca
lrwxrwxrwx 1 root root 28 Feb 5 21:23 build-dh -> /usr/share/easy-rsa/build-dh
lrwxrwxrwx 1 root root 31 Feb 5 21:23 build-inter -> /usr/share/easy-rsa/build-inter
lrwxrwxrwx 1 root root 29 Feb 5 21:23 build-key -> /usr/share/easy-rsa/build-key
lrwxrwxrwx 1 root root 34 Feb 5 21:23 build-key-pass -> /usr/share/easy-rsa/build-key-pass
lrwxrwxrwx 1 root root 36 Feb 5 21:23 build-key-pkcs12 -> /usr/share/easy-rsa/build-key-pkcs12
lrwxrwxrwx 1 root root 36 Feb 5 21:23 build-key-server -> /usr/share/easy-rsa/build-key-server
lrwxrwxrwx 1 root root 29 Feb 5 21:23 build-req -> /usr/share/easy-rsa/build-req
lrwxrwxrwx 1 root root 34 Feb 5 21:23 build-req-pass -> /usr/share/easy-rsa/build-req-pass
lrwxrwxrwx 1 root root 29 Feb 5 21:23 clean-all -> /usr/share/easy-rsa/clean-all
lrwxrwxrwx 1 root root 33 Feb 5 21:23 inherit-inter -> /usr/share/easy-rsa/inherit-inter
lrwxrwxrwx 1 root root 24 Feb 5 21:23 keys -> /usr/share/easy-rsa/keys
lrwxrwxrwx 1 root root 28 Feb 5 21:23 list-crl -> /usr/share/easy-rsa/list-crl
-rw-r--r-- 1 root root 7.7K Feb 5 21:23 openssl-0.9.6.cnf
-rw-r--r-- 1 root root 8.3K Feb 5 21:23 openssl-0.9.8.cnf
-rw-r--r-- 1 root root 8.2K Feb 5 21:23 openssl-1.0.0.cnf
-rw-r--r-- 1 root root 8.2K Feb 5 21:23 openssl.cnf
lrwxrwxrwx 1 root root 27 Feb 5 21:23 pkitool -> /usr/share/easy-rsa/pkitool
lrwxrwxrwx 1 root root 31 Feb 5 21:23 revoke-full -> /usr/share/easy-rsa/revoke-full
lrwxrwxrwx 1 root root 28 Feb 5 21:23 sign-req -> /usr/share/easy-rsa/sign-req
-rw-r--r-- 1 root root 2.1K Feb 5 21:23 vars
lrwxrwxrwx 1 root root 35 Feb 5 21:23 whichopensslcnf -> /usr/share/easy-rsa/whichopensslcnf
You can replace the openssl.cnf with whichever version you want to use for generation, I decided to go with openssl-1.0.0.cnf
Open the file ~/ca/vars and update accordingly, you can customize it however you want but you should at least modify the following to suit your organization.
export KEY_COUNTRY="US"
export KEY_PROVINCE="CA"
export KEY_CITY="SanFrancisco"
export KEY_ORG="Fort-Funston"
export KEY_EMAIL="[email protected]"
export KEY_OU="MyOrganizationalUnit"
After you have edited the file you need to source it so we have the environment variables ready, to do so you can just run
$ cd ~/ca
$ . ./vars
$ env | grep ^KEY
KEY_OU=MyOrganizationalUnit
KEY_SIZE=2048
KEY_NAME=EasyRSA
KEY_EXPIRE=3650
[email protected]
KEY_ORG=Fort-Funston
KEY_CITY=SanFrancisco
KEY_COUNTRY=US
KEY_PROVINCE=CA
KEY_CONFIG=/root/ca/openssl.cnf
KEY_DIR=/root/ca/keys
$ ./clean-all
Now you are ready to build your ca key and tls-crypt key for OpenVPN
$ cd ~/ca
$ ./build-ca
$ openvpn --genkey --secret ~/ca/keys/ta.key
You CA is ready now we have to generate a server key, make sure you sign the certificate when asked
$ cd ~/ca
$ ./build-key-server server
$ ./build-dh
And we also need to generate a client key for each client that will connect, make sure you sign the certificates when asked
$ cd ~/ca
$ ./build-key locb
Now let’s see what have we generated your keys folder should have the following files
$ ls -lh ~/ca/keys
total 84K
-rw-r--r-- 1 root root 5.4K Feb 5 21:31 01.pem
-rw-r--r-- 1 root root 5.3K Feb 5 21:35 02.pem
-rw-r--r-- 1 root root 1.7K Feb 5 21:30 ca.crt
-rw------- 1 root root 1.7K Feb 5 21:30 ca.key
-rw-r--r-- 1 root root 424 Feb 5 21:33 dh2048.pem
-rw-r--r-- 1 root root 238 Feb 5 21:35 index.txt
-rw-r--r-- 1 root root 21 Feb 5 21:35 index.txt.attr
-rw-r--r-- 1 root root 21 Feb 5 21:31 index.txt.attr.old
-rw-r--r-- 1 root root 120 Feb 5 21:31 index.txt.old
-rw-r--r-- 1 root root 5.3K Feb 5 21:35 locb.crt
-rw-r--r-- 1 root root 1.1K Feb 5 21:35 locb.csr
-rw------- 1 root root 1.7K Feb 5 21:35 locb.key
-rw-r--r-- 1 root root 3 Feb 5 21:35 serial
-rw-r--r-- 1 root root 3 Feb 5 21:31 serial.old
-rw-r--r-- 1 root root 5.4K Feb 5 21:31 server.crt
-rw-r--r-- 1 root root 1.1K Feb 5 21:31 server.csr
-rw------- 1 root root 1.7K Feb 5 21:31 server.key
-rw------- 1 root root 636 Feb 5 21:30 ta.key
Now we have everything so we can send finally do the connection between our datacenters.
Configuration
So let’s prepare some things before we start
$ mkdir -p /etc/openvpn/ccd
$ cp -v ~/ca/keys/{ca.crt,server.*,dh2048.pem,ta.key} /etc/openvpn/server
'/root/ca/keys/ca.crt' -> '/etc/openvpn/server/ca.crt'
'/root/ca/keys/server.crt' -> '/etc/openvpn/server/server.crt'
'/root/ca/keys/server.csr' -> '/etc/openvpn/server/server.csr'
'/root/ca/keys/server.key' -> '/etc/openvpn/server/server.key'
'/root/ca/keys/dh2048.pem' -> '/etc/openvpn/server/dh2048.pem'
'/root/ca/keys/ta.key' -> '/etc/openvpn/server/ta.key'
And now lets make our configuration for the server. We need to create a separate user for our process to run in, so it’s isolated from the system.
$ addgroup ovpn
$ useradd --shell=/bin/false -g ovpn ovpn
$ cat /etc/openvpn/server.conf
dev tun
persist-key
persist-local-ip
persist-remote-ip
persist-tun
topology subnet
port 1194
proto udp
keepalive 10 60
ca /etc/openvpn/server/ca.crt
cert /etc/openvpn/server/server.crt
key /etc/openvpn/server/server.key
dh /etc/openvpn/server/dh2048.pem
server 172.16.1.0 255.255.255.0
client-to-client
client-config-dir ccd
explicit-exit-notify 10
user ovpn
group ovpn
tls-crypt /etc/openvpn/server/ta.key
auth SHA512
tls-version-min 1.2
tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
ncp-ciphers AES-256-GCM:AES-256-CBC
ifconfig-pool-persist ipp.txt
status openvpn-status.log
log /var/log/openvpn.log
verb 4
route 10.0.0.0 255.255.255.0
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
reneg-sec 0
cipher AES-128-CBC
All the configuration in here is self explanatory, the only thing worth of not is route 10.0.0.0 255.255.255.0, which needs to be in the main file so the server knows that it needs to route that subnet to the connected client, if you don’t add that line then we will be missing the required routes to access the server.
Now we need to setup the vm in the location b, we still have to install the same packages as for location a, but before we do that we need to create a configuration file for the client
$ cd /etc/openvpn/ccd
cat <<EOF > /etc/openvpn/ccd/locb
push "route 10.5.100.0 255.255.252.0"
iroute 10.0.0.0 255.255.255.0
EOF
This will tell that the client locb when connected that it needs to add the route for accessing the subnet in loca and for the server to setup an internal route for the locb
Service
Let’s enable the server
$ systemctl disable openvpn
$ systemctl enable openvpn@server
$ systemctl restart openvpn@server
$ systemctl status [email protected]
VPN gateway at location B
Now we are ready to connect we need to transfer the files ca.crt,locb.*,dh2048.pem,ta.key in the keys folder from the server to the client in /etc/openvpn/client folder.
Forwarding
$ cat <<EOF > /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
$ sysctl -p /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
Installation
Let’s install the prerequisites
$ apt-get install openvpn firewalld
$ addgroup ovpn
$ useradd --shell=/bin/false -g ovpn ovpn
Firewall
Let’s configure the firewall to allow us to connect to the machine and all the necessary configuration needed for it
$ firewall-cmd --permanent --zone=public --add-source=94.x.x.x/32
$ firewall-cmd --permanent --zone=trusted --add-interface=eth1
$ firewall-cmd --permanent --zone=trusted --add-interface=tun0
$ firewall-cmd --permanent --zone=trusted --add-masquerade
$ DEV=$(ip route get 8.8.8.8 | awk 'NR==1 {print $(NF-2)}')
$ firewall-cmd --permanent --direct --passthrough ipv4 -t nat -A POSTROUTING -s 172.16.1.0/24 -o $DEV -j MASQUERADE
$ firewall-cmd --reload
Configuration
$ cat /etc/openvpn/client.conf
client
dev tun
persist-key
persist-tun
persist-local-ip
persist-remote-ip
proto udp
nobind
user ovpn
group ovpn
remote-cert-tls server
auth SHA512
verb 4
remote s2s.example.com 1194
ca /etc/openvpn/client/ca.crt
cert /etc/openvpn/client/locb.crt
key /etc/openvpn/client/locb.key
tls-crypt /etc/openvpn/client/ta.key
log /var/log/openvpn.log
keepalive 10 60
$ systemctl disable openvpn
$ systemctl enable openvpn@client
$ systemctl restart openvpn@client
$ systemctl status [email protected]
Routes
The only thing missing now is setting up the routes required so we know where we need to go for a given subnet
/etc/network/interfaces
$ cat /etc/network/interfaces
source /etc/network/interfaces.d/*
auto lo
iface lo inet loopback
auto ens18
iface ens18 inet dhcp
auto ens19
iface ens19 inet static
address 10.0.0.3
netmask 255.255.255.0
up route add -net 10.5.100.0 netmask 255.255.252.0 gw 10.0.0.5
down route add -net 10.5.100.0 netmask 255.255.252.0 gw 10.0.0.5
So now everytime our interface is configured our route will be set up so we can access the servers on the other side of the VPN
ip route
We need to add a static route on the internal network interface so we can route the subnet.
$ ip route
default via 95.x.x.x dev ens18 onlink
10.0.0.0/24 dev ens19 proto kernel scope link src 10.0.0.3
95.x.x.x/25 dev ens18 proto kernel scope link src 95.x.x.x
$ ping 10.5.100.10 -c1
PING 10.5.100.10 (10.5.100.10) 56(84) bytes of data.
--- 10.5.100.10 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
$ ip route add 10.5.100.0/22 via 10.0.0.5 dev ens19
$ ip route
default via 95.x.x.x dev ens18 onlink
10.0.0.0/24 dev ens19 proto kernel scope link src 10.0.0.3
10.5.100.0/22 via 10.0.0.5 dev ens19
95.x.x.x/25 dev ens18 proto kernel scope link src 95.x.x.x
$ ping 10.5.100.10 -c1
PING 10.5.100.10 (10.5.100.10) 56(84) bytes of data.
64 bytes from 10.5.100.10: icmp_seq=1 ttl=62 time=2.17 ms
--- 10.5.100.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.171/2.171/2.171/0.000 ms
DHCP
If we use DHCP we can easily put a stateless route so all machines automatically get the correct route.
dnsmasq
We just need to add this to the DHCP server in location A and we are done, the route will be automatically setup when the client requests an IP address.
dhcp-option=121,10.5.100.0/22,10.0.0.5
Testing
And now let’s see the result of our process finally
VPN gateway in location A
$ traceroute 10.0.0.3 -n
traceroute to 10.0.0.3 (10.0.0.3), 30 hops max, 60 byte packets
1 172.16.1.2 2.050 ms 2.038 ms 2.033 ms
2 10.0.0.3 2.239 ms 2.242 ms 2.315 ms
VPN gateway in location B
$ traceroute 10.5.100.10 -n
traceroute to 10.5.100.10 (10.5.100.10), 30 hops max, 60 byte packets
1 172.16.1.1 2.023 ms 2.007 ms 2.000 ms
2 10.5.100.10 1.991 ms 1.986 ms 1.979 ms
Machine in location B to a machine in location A
$ traceroute 10.5.100.10 -n
traceroute to 10.5.100.10 (10.5.100.10), 30 hops max, 60 byte packets
1 10.0.0.5 0.514 ms 0.497 ms 0.486 ms
2 * * *
3 10.5.100.10 2.238 ms 2.234 ms 2.219 ms
The 2nd hop is our VPN gateway doing the forwarding
Speed
The speed we got it’s not that bad, it’s 281 Mbits/sec which is around 35 MB/s. We can make it even faster by using Elliptic curve crypto (ECC) for the keys
Machine on location A
We start an iperf server on any machine on location A by logging into it and running
$ apt-get install iperf
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.5.100.10 port 5001 connected with 10.5.100.10 port 42076
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 335 MBytes 280 Mbits/sec
Machine on location B
We start an iperf client on any machine on location B by logging into it and running
$ apt-get install iperf
$ iperf -c 10.5.100.10
------------------------------------------------------------
Client connecting to 10.5.100.20, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.3 port 42076 connected with 10.5.100.20 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 335 MBytes 281 Mbits/sec