Monitor All Network Traffic On A Virtual Host Within A Virtual Machine On That Host

12 Dec 2009

Ataraxia Consulting

Recently I was asked to explore means to monitor/audit network traffic of a virtual host (i.e. all traffic on the Dom0 and DomUs) without the monitoring software running inside the privileged domain (Dom0). If you are ok with running the monitoring software inside the privileged domain then you need not read the rest of this blog.

Normal network auditing software comes in the form of tcpdump, SNORT, or even a collection of software such as OSSIM. Most of this software gleans its information by setting a network interface into promiscuous mode and then using a hub or port mirroring on your switch to duplicate all packets to that network interface for auditing. Hubs are used in lieu of switches because they operate merely as repeaters, where as a switch tries to limit network congestion by sending packets to only the port that a given destination resides behind. This same switching technology is employed by the bridge interface in Linux and most other virtualization platforms.

The most common way to achieve network auditing inside of a virtual machine guest (DomU) is to assign the physical interfaces from the bare metal to the guest (pci mapping) such that the guest sees the physical hardware, thus bypassing the bridge normally used at the Host/Hypervisor level. There are a few cons to this approach. In this deployment you can’t just add hardware and use port mirroring on your switch to capture all traffic from the Host. This will only capture traffic from sources and to destinations external to the host you’re monitoring, and won’t capture traffic among the guests running on your host. To get around that obstacle you could use this guest then as the route of all traffic to and from and among the host and virtual machine guests. However you’re then relying on a specific guest to stay available to handle the traffic and audit, and if that guest dies all other guests on the host will be without network connectivity. Some permutation of this solution is probably acceptable if you’re simply interested in monitoring traffic to and from your gateway where rogue traffic is more likely to initiate.

Another possible solution is to “simply” duplicate all packets in the privileged domain and send them “over the wire” to another virtual machine for auditing. The solution is what I’ll dub virtual port mirroring, and can be achieved with iptables (or probably more suitably ebtables) and extensions from

iptables -t mangle -A PREROUTING -j TEE --gateway

You’ll probably want to make sure this rule is at the top of your PREROUTING chain and you’re likely going to want to make the rule considerably more specific (not the least of which is preventing a feedback loop from the destination because of its own traffic). For instance, on the network I intend to deploy this on there’s a backend network handling DRBD replication, those are probably packets you’re not going to want to duplicate unless you’re extra paranoid. But this solution is simple, straight forward, and if I do say so myself elegant. Traffic in and out of the host and among the guests are all duplicated and sent to a specific destination and if that destination is down the packets merely drop. I still need to perform some stress testing to see just how far you can push the network stack before it falls over, but it’s certainly advantageous to keep the destination on the same host and not push that traffic external for security and network congestion related issues.

Note that this will not audit the rest of the traffic on your presumably switched network. You’ll want to devise a means to audit on each of your physical hosts and then collect that data in a central location later. I performed all this on the Xen stack provided from Debian in Lenny and unfortunately xtables-addons is only a Squeeze package right now (because it deps on iptables >= 1.4.3) so I ended up building these packages on my own. If there are others out there who would like to implement a similar solution in their Lenny stacks and don’t want to build the packages leave a comment and I’ll create a repo for the packages on i386 and amd64 platforms.