Multiple ISP, servers on LAN

Home Page Forums Network Management Signal a BUG Multiple ISP, servers on LAN

This topic contains 2 replies, has 0 voices, and was last updated by  atheling 8 years, 11 months ago.

Viewing 4 posts - 1 through 4 (of 4 total)
  • Author
    Posts
  • #42066

    atheling
    Member

    Please excuse my long post. Also, this is my first serious experince with iptables and ip rules. So I might be entirely wrong about the issue.

    The environment:
    I have a Soekris net5501 with an extra NIC interface giving a total of five ethernet interfaces. ETH00 and ETH01 are connected to two different ISPs. ETH00 is connected to a DSL modem that uses PPPoE while ETH01 has a fixed IP. All the other interfaces connect to different local LAN networks.

    Looking at the output of “ip rules” and “ip route”, I see that Zeroshell has setup routing realm 101 (0x65) for one ISP and realm 102 (0x66) for the other (at the moment I’m setup for failover but I recall loadbalance was the same):


    root@zeroshell root> ip rule list
    0: from all lookup local
    32764: from all fwmark 0x66 lookup 102
    32765: from all fwmark 0x65 lookup 101
    32766: from all lookup main
    32767: from all lookup default
    root@zeroshell root> ip route list table 101
    nn.nn.nn.180/30 dev ETH01 proto kernel scope link src nn.nn.nn.181
    10.7.52.0/24 dev ETH04 proto kernel scope link src 10.7.52.1
    10.7.53.0/24 dev ETH03 proto kernel scope link src 10.7.53.1
    10.7.54.0/24 dev ETH02 proto kernel scope link src 10.7.54.1
    192.168.250.0/24 dev VPN99 proto kernel scope link src 192.168.250.254
    10.4.27.0/24 dev ETH00 proto kernel scope link src 10.4.27.1
    default via nn.nn.nn.182 dev ETH01
    unreachable default metric 99
    root@zeroshell root> ip route list table 102
    nn.nn.nn.180/30 dev ETH01 proto kernel scope link src nn.nn.nn.181
    10.7.52.0/24 dev ETH04 proto kernel scope link src 10.7.52.1
    10.7.53.0/24 dev ETH03 proto kernel scope link src 10.7.53.1
    10.7.54.0/24 dev ETH02 proto kernel scope link src 10.7.54.1
    192.168.250.0/24 dev VPN99 proto kernel scope link src 192.168.250.254
    10.4.27.0/24 dev ETH00 proto kernel scope link src 10.4.27.1
    default dev ppp0 scope link
    unreachable default metric 99
    root@zeroshell root>

    Looking at the mangle tables, ZS takes the routing realm for a new connection and saves it with the connection. When a new packet is received it restores the mark from connection to the packet:


    Chain PREROUTING (policy ACCEPT 319K packets, 200M bytes)
    pkts bytes target prot opt in out source destination
    319K 200M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
    319K 200M NetBalancer all -- * * 0.0.0.0/0 0.0.0.0/0

    Chain POSTROUTING (policy ACCEPT 316K packets, 200M bytes)
    pkts bytes target prot opt in out source destination
    31739 3511K NB_CT_POST all -- * * 0.0.0.0/0 0.0.0.0/0 state NEW
    316K 200M NB_STAT all -- * * 0.0.0.0/0 0.0.0.0/0

    Chain NB_CT_POST (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 realm 0x66 MARK set 0x66
    7895 591K MARK all -- * * 0.0.0.0/0 0.0.0.0/0 realm 0x65 MARK set 0x65
    31739 3511K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save

    The end result of all this is that for connections initiated within the LAN, the intial routing decision is recorded and subsequent packets for that connection will use the same external interface. This is good.

    The problem is when you have inbound traffic. I host a mail and web server, available through either link. My two links are quite different on speeds (the cable modem over ETH01 is 10x faster than my DSL) or if I am doing fail over the default route will always be over ETH01 for new connections. So traffic coming in over ppp0 will be responded to over ETH01.

    The fix for this is to set the routing preference for new connections from the ISPs in the mangle prerouting chain. I’ve done it here by adding my own custom chain:


    Chain PREROUTING (policy ACCEPT 319K packets, 200M bytes)
    pkts bytes target prot opt in out source destination
    319K 200M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
    319K 200M NetBalancer all -- * * 0.0.0.0/0 0.0.0.0/0
    15687 2119K custom_preroute all -- * * 0.0.0.0/0 0.0.0.0/0 state NEW

    Chain custom_preroute (1 references)
    pkts bytes target prot opt in out source destination
    1963 113K MARK all -- ETH01 * 0.0.0.0/0 0.0.0.0/0 MARK set 0x65
    1963 113K CONNMARK all -- ETH01 * 0.0.0.0/0 0.0.0.0/0 CONNMARK save
    1630 79635 MARK all -- ppp0 * 0.0.0.0/0 0.0.0.0/0 MARK set 0x66
    1630 79635 CONNMARK all -- ppp0 * 0.0.0.0/0 0.0.0.0/0 CONNMARK save

    Before I created this custom mangle prerouting rule attempts to connect via the backup link failed. With these rules it works perfectly.

    It looks like the nb_setautomarking script would be the place to set this up. But when I explore the contents of /var/register/system/net/nb/Gateways/01/Interface I don’t see ETH01 (I do see ppp0 in Gateways/02/Interface). So it is not clear to me where the correct location is to get the interface name needed to generate these rules.

    There is an issue in the QoS table because it uses the same marking system as this routing. The above fix for inbound connections from the backup ISP exacerbates the QOS bug. But that is a topic of another bug which I’ll post separately in a little while.

    #49167

    imported_fulvio
    Participant

    Thanks a lot for the suggestion. Sureli I’ll include it in the next release.
    QoS and load balancer/failover cannot work simultaneously. You should use separately boxes.

    Regards
    Fulvio

    #49168

    RDelpopolo
    Member

    Can you explain easily how to solve that problem?

    #49169

    atheling
    Member

    @rdelpopolo wrote:

    Can you explain easily how to solve that problem?

    I’ve posted a patch that works for me and for some others. It did not get included in beta13 but the patch should work on that release too. See http://www.zeroshell.net/eng/forum/viewtopic.php?p=7600#7600 for more details.

Viewing 4 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.