Site to site with OpenVPN with TLS and FirewallD on Debian

So I needed to setup a site to site transport between our two datacenters, so our internal network will be able to communicate between the two datacenters easily.

There are a lot of ways to setup site to site VPN link between two networks, in our case let’s take a look at one way using OpenVPN.

This is quite long tutorial so get yourself a bit comfortable so we can get started.

Our network looks like this

Site to site VPN

Datacenter at location A has internal network 10.5.200.20 on the interface eth1 and 94.x.x.x.x on eth0, we will configure the openvpn server on this location. Datacenter at location B has internal network 10.0.0.5 on the interface eth1 and 93.x.x.x.x on eth0.

Both machine have two network interfaces eth0 which is connected to the internet, and eth1 which is connected the the LAN

The IP subnet has to be different for this to work, if they are on the same subnet this won’t work.

VPN gateway at location A

So to begin login to the machine in the location A

Forwarding

$ cat <<EOF > /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
$ sysctl -p /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

Installation

apt-get install openvpn easy-rsa firewalld

Firewall

Let’s configure the firewall to allow us to connect to the machine and all the necessary configuration needed for it

$ firewall-cmd --permanent --zone=public --add-source=93.x.x.x/32
$ firewall-cmd --permanent --zone=trusted --add-interface=eth1
$ firewall-cmd --permanent --zone=trusted --add-interface=tun0
$ firewall-cmd --permanent --zone=trusted --add-masquerade
$ DEV=$(ip route get 8.8.8.8 | awk 'NR==1 {print $(NF-2)}')
$ firewall-cmd --permanent --direct --passthrough ipv4 -t nat -A POSTROUTING -s  172.16.1.0/24 -o $DEV -j MASQUERADE
$ firewall-cmd --reload

Generating OpenVPN keys

After we have installed all this we need to generate some configuration and tls keys for us to use.

$ make-cadir ~/ca
$ cd ~/ca
$ ls -lah
drwx------ 2 root root 4.0K Feb  5 21:23 .
drwxr-xr-x 4 root root 4.0K Feb  5 21:23 ..
lrwxrwxrwx 1 root root   28 Feb  5 21:23 build-ca -> /usr/share/easy-rsa/build-ca
lrwxrwxrwx 1 root root   28 Feb  5 21:23 build-dh -> /usr/share/easy-rsa/build-dh
lrwxrwxrwx 1 root root   31 Feb  5 21:23 build-inter -> /usr/share/easy-rsa/build-inter
lrwxrwxrwx 1 root root   29 Feb  5 21:23 build-key -> /usr/share/easy-rsa/build-key
lrwxrwxrwx 1 root root   34 Feb  5 21:23 build-key-pass -> /usr/share/easy-rsa/build-key-pass
lrwxrwxrwx 1 root root   36 Feb  5 21:23 build-key-pkcs12 -> /usr/share/easy-rsa/build-key-pkcs12
lrwxrwxrwx 1 root root   36 Feb  5 21:23 build-key-server -> /usr/share/easy-rsa/build-key-server
lrwxrwxrwx 1 root root   29 Feb  5 21:23 build-req -> /usr/share/easy-rsa/build-req
lrwxrwxrwx 1 root root   34 Feb  5 21:23 build-req-pass -> /usr/share/easy-rsa/build-req-pass
lrwxrwxrwx 1 root root   29 Feb  5 21:23 clean-all -> /usr/share/easy-rsa/clean-all
lrwxrwxrwx 1 root root   33 Feb  5 21:23 inherit-inter -> /usr/share/easy-rsa/inherit-inter
lrwxrwxrwx 1 root root   24 Feb  5 21:23 keys -> /usr/share/easy-rsa/keys
lrwxrwxrwx 1 root root   28 Feb  5 21:23 list-crl -> /usr/share/easy-rsa/list-crl
-rw-r--r-- 1 root root 7.7K Feb  5 21:23 openssl-0.9.6.cnf
-rw-r--r-- 1 root root 8.3K Feb  5 21:23 openssl-0.9.8.cnf
-rw-r--r-- 1 root root 8.2K Feb  5 21:23 openssl-1.0.0.cnf
-rw-r--r-- 1 root root 8.2K Feb  5 21:23 openssl.cnf
lrwxrwxrwx 1 root root   27 Feb  5 21:23 pkitool -> /usr/share/easy-rsa/pkitool
lrwxrwxrwx 1 root root   31 Feb  5 21:23 revoke-full -> /usr/share/easy-rsa/revoke-full
lrwxrwxrwx 1 root root   28 Feb  5 21:23 sign-req -> /usr/share/easy-rsa/sign-req
-rw-r--r-- 1 root root 2.1K Feb  5 21:23 vars
lrwxrwxrwx 1 root root   35 Feb  5 21:23 whichopensslcnf -> /usr/share/easy-rsa/whichopensslcnf

You can replace the openssl.cnf with whichever version you want to use for generation, I decided to go with openssl-1.0.0.cnf

Open the file ~/ca/vars and update accordingly, you can customize it however you want but you should at least modify the following to suit your organization.

export KEY_COUNTRY="US"
export KEY_PROVINCE="CA"
export KEY_CITY="SanFrancisco"
export KEY_ORG="Fort-Funston"
export KEY_EMAIL="[email protected]"
export KEY_OU="MyOrganizationalUnit"

After you have edited the file you need to source it so we have the environment variables ready, to do so you can just run

$ cd ~/ca
$ . ./vars
$ env | grep ^KEY
KEY_OU=MyOrganizationalUnit
KEY_SIZE=2048
KEY_NAME=EasyRSA
KEY_EXPIRE=3650
[email protected]
KEY_ORG=Fort-Funston
KEY_CITY=SanFrancisco
KEY_COUNTRY=US
KEY_PROVINCE=CA
KEY_CONFIG=/root/ca/openssl.cnf
KEY_DIR=/root/ca/keys
$ ./clean-all

Now you are ready to build your ca key and tls-crypt key for OpenVPN

$ cd ~/ca
$ ./build-ca
$ openvpn --genkey --secret ~/ca/keys/ta.key

You CA is ready now we have to generate a server key, make sure you sign the certificate when asked

$ cd ~/ca
$ ./build-key-server server
$ ./build-dh

And we also need to generate a client key for each client that will connect, make sure you sign the certificates when asked

$ cd ~/ca
$ ./build-key locb

Now let’s see what have we generated your keys folder should have the following files

$ ls -lh ~/ca/keys
total 84K
-rw-r--r-- 1 root root 5.4K Feb  5 21:31 01.pem
-rw-r--r-- 1 root root 5.3K Feb  5 21:35 02.pem
-rw-r--r-- 1 root root 1.7K Feb  5 21:30 ca.crt
-rw------- 1 root root 1.7K Feb  5 21:30 ca.key
-rw-r--r-- 1 root root  424 Feb  5 21:33 dh2048.pem
-rw-r--r-- 1 root root  238 Feb  5 21:35 index.txt
-rw-r--r-- 1 root root   21 Feb  5 21:35 index.txt.attr
-rw-r--r-- 1 root root   21 Feb  5 21:31 index.txt.attr.old
-rw-r--r-- 1 root root  120 Feb  5 21:31 index.txt.old
-rw-r--r-- 1 root root 5.3K Feb  5 21:35 locb.crt
-rw-r--r-- 1 root root 1.1K Feb  5 21:35 locb.csr
-rw------- 1 root root 1.7K Feb  5 21:35 locb.key
-rw-r--r-- 1 root root    3 Feb  5 21:35 serial
-rw-r--r-- 1 root root    3 Feb  5 21:31 serial.old
-rw-r--r-- 1 root root 5.4K Feb  5 21:31 server.crt
-rw-r--r-- 1 root root 1.1K Feb  5 21:31 server.csr
-rw------- 1 root root 1.7K Feb  5 21:31 server.key
-rw------- 1 root root  636 Feb  5 21:30 ta.key

Now we have everything so we can send finally do the connection between our datacenters.

Configuration

So let’s prepare some things before we start

$ mkdir -p /etc/openvpn/ccd
$ cp -v ~/ca/keys/{ca.crt,server.*,dh2048.pem,ta.key} /etc/openvpn/server
'/root/ca/keys/ca.crt' -> '/etc/openvpn/server/ca.crt'
'/root/ca/keys/server.crt' -> '/etc/openvpn/server/server.crt'
'/root/ca/keys/server.csr' -> '/etc/openvpn/server/server.csr'
'/root/ca/keys/server.key' -> '/etc/openvpn/server/server.key'
'/root/ca/keys/dh2048.pem' -> '/etc/openvpn/server/dh2048.pem'
'/root/ca/keys/ta.key' -> '/etc/openvpn/server/ta.key'

And now lets make our configuration for the server. We need to create a separate user for our process to run in, so it’s isolated from the system.

$ addgroup ovpn
$ useradd --shell=/bin/false -g ovpn ovpn
$ cat /etc/openvpn/server.conf
dev tun
persist-key
persist-local-ip
persist-remote-ip
persist-tun
topology subnet
port 1194
proto udp
keepalive 10 60

ca /etc/openvpn/server/ca.crt
cert /etc/openvpn/server/server.crt
key /etc/openvpn/server/server.key
dh /etc/openvpn/server/dh2048.pem

server 172.16.1.0 255.255.255.0
client-to-client

client-config-dir ccd

explicit-exit-notify 10

user ovpn
group ovpn

tls-crypt /etc/openvpn/server/ta.key
auth SHA512
tls-version-min 1.2
tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
ncp-ciphers AES-256-GCM:AES-256-CBC

ifconfig-pool-persist ipp.txt
status openvpn-status.log
log /var/log/openvpn.log
verb 4

route 10.0.0.0 255.255.255.0

tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
reneg-sec 0

cipher AES-128-CBC

All the configuration in here is self explanatory, the only thing worth of not is route 10.0.0.0 255.255.255.0, which needs to be in the main file so the server knows that it needs to route that subnet to the connected client, if you don’t add that line then we will be missing the required routes to access the server.

Now we need to setup the vm in the location b, we still have to install the same packages as for location a, but before we do that we need to create a configuration file for the client

$ cd /etc/openvpn/ccd
cat <<EOF > /etc/openvpn/ccd/locb
push "route 10.5.100.0 255.255.252.0"
iroute 10.0.0.0 255.255.255.0
EOF

This will tell that the client locb when connected that it needs to add the route for accessing the subnet in loca and for the server to setup an internal route for the locb

Service

Let’s enable the server

$ systemctl disable openvpn
$ systemctl enable openvpn@server
$ systemctl restart openvpn@server
$ systemctl status [email protected]

VPN gateway at location B

Now we are ready to connect we need to transfer the files ca.crt,locb.*,dh2048.pem,ta.key in the keys folder from the server to the client in /etc/openvpn/client folder.

Forwarding

$ cat <<EOF > /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF
$ sysctl -p /etc/sysctl.d/30-openvpn-forward.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

Installation

Let’s install the prerequisites

$ apt-get install openvpn firewalld
$ addgroup ovpn
$ useradd --shell=/bin/false -g ovpn ovpn

Firewall

Let’s configure the firewall to allow us to connect to the machine and all the necessary configuration needed for it

$ firewall-cmd --permanent --zone=public --add-source=94.x.x.x/32
$ firewall-cmd --permanent --zone=trusted --add-interface=eth1
$ firewall-cmd --permanent --zone=trusted --add-interface=tun0
$ firewall-cmd --permanent --zone=trusted --add-masquerade
$ DEV=$(ip route get 8.8.8.8 | awk 'NR==1 {print $(NF-2)}')
$ firewall-cmd --permanent --direct --passthrough ipv4 -t nat -A POSTROUTING -s  172.16.1.0/24 -o $DEV -j MASQUERADE
$ firewall-cmd --reload

Configuration

$ cat /etc/openvpn/client.conf
client
dev tun
persist-key
persist-tun
persist-local-ip
persist-remote-ip
proto udp
nobind
user ovpn
group ovpn
remote-cert-tls server
auth SHA512
verb 4

remote s2s.example.com 1194

ca /etc/openvpn/client/ca.crt
cert /etc/openvpn/client/locb.crt
key /etc/openvpn/client/locb.key
tls-crypt /etc/openvpn/client/ta.key

log /var/log/openvpn.log

keepalive 10 60
$ systemctl disable openvpn
$ systemctl enable openvpn@client
$ systemctl restart openvpn@client
$ systemctl status [email protected]

Routes

The only thing missing now is setting up the routes required so we know where we need to go for a given subnet

/etc/network/interfaces

$ cat /etc/network/interfaces
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto ens18
iface ens18 inet dhcp

auto ens19
iface ens19 inet static
    address 10.0.0.3
    netmask 255.255.255.0
    up route add -net 10.5.100.0 netmask 255.255.252.0 gw 10.0.0.5
    down route add -net 10.5.100.0 netmask 255.255.252.0 gw 10.0.0.5

So now everytime our interface is configured our route will be set up so we can access the servers on the other side of the VPN

ip route

We need to add a static route on the internal network interface so we can route the subnet.

$ ip route
default via 95.x.x.x dev ens18 onlink
10.0.0.0/24 dev ens19 proto kernel scope link src 10.0.0.3
95.x.x.x/25 dev ens18 proto kernel scope link src 95.x.x.x

$ ping 10.5.100.10 -c1
PING 10.5.100.10 (10.5.100.10) 56(84) bytes of data.

--- 10.5.100.10 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

$ ip route add 10.5.100.0/22 via 10.0.0.5 dev ens19

$ ip route
default via 95.x.x.x dev ens18 onlink
10.0.0.0/24 dev ens19 proto kernel scope link src 10.0.0.3
10.5.100.0/22 via 10.0.0.5 dev ens19
95.x.x.x/25 dev ens18 proto kernel scope link src 95.x.x.x

$ ping 10.5.100.10 -c1
PING 10.5.100.10 (10.5.100.10) 56(84) bytes of data.
64 bytes from 10.5.100.10: icmp_seq=1 ttl=62 time=2.17 ms

--- 10.5.100.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.171/2.171/2.171/0.000 ms

DHCP

If we use DHCP we can easily put a stateless route so all machines automatically get the correct route.

dnsmasq

We just need to add this to the DHCP server in location A and we are done, the route will be automatically setup when the client requests an IP address.

dhcp-option=121,10.5.100.0/22,10.0.0.5

Testing

And now let’s see the result of our process finally

VPN gateway in location A

$ traceroute 10.0.0.3 -n
traceroute to 10.0.0.3 (10.0.0.3), 30 hops max, 60 byte packets
 1  172.16.1.2  2.050 ms  2.038 ms  2.033 ms
 2  10.0.0.3  2.239 ms  2.242 ms  2.315 ms

VPN gateway in location B

$ traceroute 10.5.100.10 -n
traceroute to 10.5.100.10 (10.5.100.10), 30 hops max, 60 byte packets
 1  172.16.1.1  2.023 ms  2.007 ms  2.000 ms
 2  10.5.100.10  1.991 ms  1.986 ms  1.979 ms

Machine in location B to a machine in location A

$ traceroute 10.5.100.10 -n
traceroute to 10.5.100.10 (10.5.100.10), 30 hops max, 60 byte packets
 1  10.0.0.5  0.514 ms  0.497 ms  0.486 ms
 2  * * *
 3  10.5.100.10  2.238 ms  2.234 ms  2.219 ms

The 2nd hop is our VPN gateway doing the forwarding

Speed

The speed we got it’s not that bad, it’s 281 Mbits/sec which is around 35 MB/s. We can make it even faster by using Elliptic curve crypto (ECC) for the keys

Machine on location A

We start an iperf server on any machine on location A by logging into it and running

$ apt-get install iperf
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.5.100.10 port 5001 connected with 10.5.100.10 port 42076
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   335 MBytes   280 Mbits/sec

Machine on location B

We start an iperf client on any machine on location B by logging into it and running

$ apt-get install iperf
$ iperf -c 10.5.100.10
------------------------------------------------------------
Client connecting to 10.5.100.20, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.3 port 42076 connected with 10.5.100.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   335 MBytes   281 Mbits/sec