Please forgive my long post, I’ll try to break it up into pieces so that you can skip over sections that are not of interest to you.
The updated patch posted by mrevjd
With the exception of changes in failoverd and new changes in nb_testfo, mrevjd’s patch matches what I have in my beta14 working area. So if you are just looking for QoS fixes and having incoming traffic to servers exit on the appropriate interface you should be fine with his patch. Or, for that matter, one of my older patches.
I am still running beta 12 but have updated my patch for the beta 14 release and am actually running this proposed beta 14 patch on my beta 12 system. It may work for you but is untested on a beta 14 system. You can download it from:
It has significant changes in failoverd, including a new concept of how to detect failed links.
When should you use net balancing
If you can possibly use a bonded interface to handle link failover and/or load balancing, I strongly suggest you do: The Linux bonding driver is robust, reasonably well documented and well supported. It can load balance traffic on a single TCP connection which net balancing can’t do.
However bonding requires that both ends (you and your ISP) are set up for it. That may not be the case.
If you are putting in links between offices, you could setup an OpenVPN link with it associated pseudo device on each physical link then bond those for higher throughput between offices. You would not need to coordinate with your ISP for that. I believe the documentation for this resides in these forums and on the main Zeroshell web site.
What net balancing attempts to do and its limitations
If, like me, you have multiple ISPs using different technologies to provide redundant Internet access then you are dealing with WAN links that cannot be bonded.
Net balancing attempts to balance traffic at the connection level (all datagrams in the same connection go through the one interface): Any one TCP connection will be bound to a specific interface. More over, under many, perhaps most, situations all successive TCP connections to a specific IP address will be bound to that same interface.
This “stickiness” where successive TCP connections to the same IP address use the same gateway is due to the Linux kernel finding the route in its routing cache. And for general use it is a good thing because otherwise you’d never be able to log into a site that uses HTTPS.
If an WAN link goes down, then the normal routing entries that allow user traffic to go through that WAN’s interface are removed from the routing tables. This means that marking an interface as failed could remove the ability to ping through it to detect that the link has been restored.
Linux routing background information
Sending datagrams for a connection to a specific interface is done through the marking of datagrams with tags, called fwmark in the routing area and under a different name in the iptables area. Basically iptables examine each datagram, looks at a connection table and other information then tags the datagram with a fwmark to tell the router how to handle it.
The routing cache, the routing policy database (RPDB) (a.k.a ip rules) and one or more routing tables are then used to send the datagram on its way through the appropriate interface.
At this point you may want to go read the information on how Linux routing works at http://linux-ip.net/html/routing-selection.html. In particular you will want to look at “Example 4.4. Routing Selection Algorithm in Pseudo-code”.
The task performed by failoverd
failoverd has a conceptually easy job: Ping a set of destination IP addresses over each WAN (gateway) link. If the link is up and the pings fail then bring the interface down. If the link is down and the pings succeed then bring the interface up.
To do that failoverd has to send pings through the Linux routing logic so they are sent through specific interfaces. Reading the man pages for ping indicates you can specify the interface, but when you attempt that on a downed interface you will never succeed so you can never bring the interface up. So forget the easy way to do things…
All of the versions of failoverd I’ve seen released in Zeroshell rely on ping’s ability to set a TOS flag on each datagram. Before a ping is send a rule is added to the routing policy database that directs the routing logic to send datagrams with that TOS flag to the routing table for the specific gateway. The default route in that table then is used to send the datagram on its way.
However it seems that every time you insert or remove a rule in the routing policy database Linux appears to flush its routing cache. Therein lies a problem. With the cache flushed, the “stickiness” for TCP connections is reset and any transactions that assume the IP address is constant from one TCP connection to the next will fail (e.g. HTTPS to a bank web site).
This is my issue with the stock release of failoverd. If this is not a problem for you, then you need not use a modified version of failoverd.
But if you need HTTPS to work a different method of directing pings to the desired WAN link is needed. One that won’t clear the routing cache.
There is another issue: When a gateway is marked as failed the default route entry for it is removed from the normal (main and gateway specific) tables. So even if we can direct the ping to the gateway specific table it won’t go out the correct interface. I’m not sure how the original failoverd handled this as it appears this issue is in that code too.
How this version of failoverd handles ping routing
The solution this patch version of failoverd uses is to add some failoverd specific routing tables that are not re-written by the normal interface and traffic shaping logic. The tables are simple with one entry listing the default route. And then one ip rule per gateway is added using a fwmark different than the normal traffic categorization uses. The routing rules and tables are set up only when something changes (gateway added/removed, failoverd restarted, etc.) but not for each ping cycle.
When a ping is to be sent a new iptable entry is used to mark pings from the router for the desired routing table then the ping(s) are send. Changing the iptables on a per ping gateway change does not appear to affect the routing cache so the connection to connection stickyness needed for HTTPS sessions is maintained.
When an interface is marked down, the routing table used by failoverd for pings should not be affected so pings can still be sent through a failed gateway.
Issues and questionable logic
failoverd does not check gateways if their default route in their gateway specific routing table does not exist. I think this check can be removed with the new logic and should make it more robust.
Tied into this, I’ve noticed that the pppoe interface does not always add its default route back into the routing tables. The result in ad hoc testing is that sometimes the ppp routes do not come back up. I’ve seen this perhaps 10% of the time. I suspect that removing the default route check in failoverd might paper over that issue.
I think this patch should work on beta 12, beta 13 and beta 14.
This patch does a lot better job than my previous one that rolled over and died if the ppp0 connection died when the ETH connection was disabled or failed.
If you feel adventurous please give it a try and let me know how it goes. If you provide feedback please also give some indication of your network topology (number of gateways, type (pppoe, etc.).