matoski.com

I needed to forward some ports from multiple KVM machines, I tried with iptables, but the problem is libvirt adds some rules of it’s own, and the rules were never in the correct place so it didn’t work.

Fortunately KVM supports hooks, and we can use them to do what we need.

You should be able to easily adapt this to any linux distro.

• /etc/libvirt/hooks/daemon
  

Executed when the libvirt daemon is started, stopped, or reloads its configuration

• /etc/libvirt/hooks/qemu

Executed when a QEMU guest is started, stopped, or migrated

• /etc/libvirt/hooks/lxc
  

Executed when an LXC guest is started or stopped

For a more detailed list we you can check libvirt hooks on the libvirt home page.

For us we will need to use /etc/libvirt/hooks/qemu script, because that is the script which triggers whenever we have VM going through the following states

• prepare (libvirt 0.9.0)
• start (libvirt 0.8.0)
• started (libvirt 0.9.13)
• stopped (libvirt 0.8.0)
• release (libvirt 0.9.0)
• migrate (libvirt 0.9.11)
• reconnect (libvirt 0.9.13)
• attach (libvirt 0.9.13)

In our case we are interested in three events, and we will split them in two groups

• stopped or reconnect
• start or reconnect

We want to automatically add/remove rules to iptables on starting/stopping the VM, let’s see a structure we are gonna use to hold the mappings.

{
  '<guest name (as defined in xml)>': {
     'ip': '<private ip>',
     'publicip': '<public ip>',
     'portmap': 'all' |
            {
               '<proto>': [(<host port>, <guest port>), ...], ...
            }
     }, ...
}

From the definition it is obvious what should go where

• proto is the protocol
• portmap can be either all or the map

Also we should log all the actions taken by this script to syslog if we want to do checking or debuging, to see if everything is properly working.

I would gladly accept any improvements to this scripts, so feel free to improve this script, and I will update it, on the page. This script has been tested on Python 2.7.3 and should work on Python 3, as for older versions I don’t know.

So let’s take a look at the python script located under /etc/libvirt/hooks/qemu, make sure you execute chmod +x /etc/libvirt/hooks/qemu

#!/usr/bin/python
# -*- coding: utf-8 -*-

import sys
import os
import syslog

domain = sys.argv[1]
action = sys.argv[2]

iptables = '/sbin/iptables'

mapping = {
  "vm1": {
    "ip": "192.168.22.1",
    "publicip": "x.x.x.x",
    "portmap": {
      "tcp": [
        (3901, 80),
        (3701, 22),
        (31621, 31621)
      ]
    }
  }
}

syslog.syslog('Processing QEMU Signal, domain: ' + domain + ', action:'
              + action)

def rules(act, map_dict):
    if map_dict['portmap'] == 'all':
        cmd = \
            '{} -t nat {} PREROUTING -d {} -j DNAT --to {}'.format(iptables,
                act, map_dict['publicip'], map_dict['ip'])
        os.system(cmd)
        syslog.syslog(cmd)
        cmd = \
            '{} -t nat {} POSTROUTING -s {} -j SNAT --to {}'.format(iptables,
                act, map_dict['ip'], map_dict['publicip'])
        os.system(cmd)
        syslog.syslog(cmd)
        cmd = \
            '{} -t filter {} FORWARD -d {} -j ACCEPT'.format(iptables,
                act, map_dict['ip'])
        os.system(cmd)
        syslog.syslog(cmd)
        cmd = \
            '{} -t filter {} FORWARD -s {} -j ACCEPT'.format(iptables,
                act, map_dict['ip'])
        os.system(cmd)
        syslog.syslog(cmd)
    else:
        for proto in map_dict['portmap']:
            for portmap in map_dict['portmap'].get(proto):
                cmd = \
                    '{} -t nat {} PREROUTING -d {} -p {} --dport {} -j DNAT --to {}:{}'.format(
                    iptables,
                    act,
                    map_dict['publicip'],
                    proto,
                    str(portmap[0]),
                    map_dict['ip'],
                    str(portmap[1]),
                    )
                os.system(cmd)
                syslog.syslog(cmd)
                cmd = \
                    '{} -t filter {} FORWARD -d {} -p {} --dport {} -j ACCEPT'.format(iptables,
                        act, map_dict['ip'], proto, str(portmap[1]))
                os.system(cmd)
                syslog.syslog(cmd)
                cmd = \
                    '{} -t filter {} FORWARD -s {} -p {} --sport {} -j ACCEPT'.format(iptables,
                        act, map_dict['ip'], proto, str(portmap[1]))
                os.system(cmd)
                syslog.syslog(cmd)


host = mapping.get(domain)

if host is None:
    sys.exit(0)

if action == 'stopped' or action == 'reconnect':
    rules('-D', host)
    syslog.syslog('Removed all the iptables rules for ' + domain)

if action == 'start' or action == 'reconnect':
    rules('-I', host)
    syslog.syslog('Created all the iptables rules for ' + domain)

I also added this to my /etc/sysctl.conf

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
net.ipv6.conf.default.autoconf=0
net.ipv6.conf.default.accept_dad=0
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.default.accept_ra_defrtr=0
net.ipv6.conf.default.accept_ra_rtr_pref=0
net.ipv6.conf.default.accept_ra_pinfo=0
net.ipv6.conf.default.accept_source_route=0
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.default.forwarding=0
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.all.accept_dad=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.all.accept_ra_defrtr=0
net.ipv6.conf.all.accept_ra_rtr_pref=0
net.ipv6.conf.all.accept_ra_pinfo=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.all.forwarding=0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.log_martians = 0
net.ipv4.ip_local_port_range = 32768 61000
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 2400
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0

If you want you can add this too, but if you don’t want to you really need to add net.ipv4.ip_forward.

After editing /etc/sysctl.conf you need to issue

sysctl -p

To initialize and read the settings from /etc/sysctl.conf

Now you need to restart the libvirt service so it will initialize the hook, otherwise it will not call it.

libvirt Adds it’s own rules to ip tables, so if you want to do this manually you will have to make sure you add the correct rules for this to work.

For my setup the iptables was as follows, the rules will be automatically added/removed depending on the state of the machine whether it is running or not

# Generated by iptables-save v1.4.14 on Sat Dec 14 18:31:26 2013
*nat
:PREROUTING ACCEPT [126:7694]
:INPUT ACCEPT [126:7694]
:OUTPUT ACCEPT [223:17030]
:POSTROUTING ACCEPT [223:17030]
-A POSTROUTING -s 192.168.22.0/24 ! -d 192.168.22.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.22.0/24 ! -d 192.168.22.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.22.0/24 ! -d 192.168.22.0/24 -j MASQUERADE
COMMIT
# Completed on Sat Dec 14 18:31:26 2013
# Generated by iptables-save v1.4.14 on Sat Dec 14 18:31:26 2013
*mangle
:PREROUTING ACCEPT [12485:1642751]
:INPUT ACCEPT [12485:1642751]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [12074:3717410]
:POSTROUTING ACCEPT [12074:3717410]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Sat Dec 14 18:31:26 2013
# Generated by iptables-save v1.4.14 on Sat Dec 14 18:31:26 2013
*filter
:INPUT ACCEPT [12466:1639919]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [12074:3717410]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d 192.168.22.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.22.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Sat Dec 14 18:31:26 2013