Heinzelmännchen Problem

[Pit]

nb_fw.rej:

***************
*** 1,24 ****
#!/bin/sh
. /etc/kerbynet.conf
iptables -t mangle -D PREROUTING -j CONNMARK –restore-mark 2>/dev/null
– iptables -t mangle -D PREROUTING -j NetBalancer 2>/dev/null
– iptables -t mangle -D INPUT -j NetBalancer 2>/dev/null
– iptables -t mangle -D OUTPUT -j NetBalancer 2>/dev/null
iptables -t mangle -D OUTPUT -j OpenVPN 2>/dev/null
iptables -t mangle -D POSTROUTING -m state –state NEW -j NB_CT_POST 2>/dev/null
iptables -t mangle -D POSTROUTING -j NB_STAT 2>/dev/null
if [ “`cat $REGISTER/system/net/nb/Enabled 2>/dev/null`” = yes ] ; then
iptables -t mangle -I PREROUTING 1 -j CONNMARK –restore-mark
– iptables -t mangle -I PREROUTING 2 -j NetBalancer
iptables -t mangle -I POSTROUTING 1 -m state –state NEW -j NB_CT_POST 2>/dev/null
iptables -t mangle -I POSTROUTING 2 -j NB_STAT 2>/dev/null
– iptables -t mangle -I INPUT 1 -j NetBalancer
– iptables -t mangle -I OUTPUT 1 -j NetBalancer
– iptables -t mangle -I OUTPUT 2 -j OpenVPN
fi
$SCRIPTS/nb_vpn 2> /dev/null
$SCRIPTS/nb_setautomarking 2>/dev/null
–
–
–

— 1,35 —-
#!/bin/sh
. /etc/kerbynet.conf
iptables -t mangle -D PREROUTING -j CONNMARK –restore-mark 2>/dev/null
+ iptables -t mangle -D PREROUTING -m state –state NEW -j NB_CT_PRE 2>/dev/null
+ iptables -t mangle -D PREROUTING -m state –state NEW -j NetBalancer 2>/dev/null
+ iptables -t mangle -D INPUT -m state –state NEW -j NB_CT_POST 2>/dev/null
+ iptables -t mangle -D OUTPUT -j CONNMARK –restore-mark 2>/dev/null
+ iptables -t mangle -D OUTPUT -m state –state NEW -j NB_FO_PRE 2>/dev/null
+ iptables -t mangle -D OUTPUT -m state –state NEW -j NetBalancer 2>/dev/null
iptables -t mangle -D OUTPUT -j OpenVPN 2>/dev/null
iptables -t mangle -D POSTROUTING -m state –state NEW -j NB_CT_POST 2>/dev/null
iptables -t mangle -D POSTROUTING -j NB_STAT 2>/dev/null
+ # Need QoS to be done in mangle POSTROUTING. Note that if NetBalance
+ # is enabled then we will insert those rules/chains first. So any
+ # routing marks will be handled before we blow them away with QoS
+ # marks.
+ iptables -t mangle -D POSTROUTING -j QoS 2>/dev/null
+ iptables -t mangle -I POSTROUTING 1 -j QoS 2>/dev/null
if [ “`cat $REGISTER/system/net/nb/Enabled 2>/dev/null`” = yes ] ; then
iptables -t mangle -I PREROUTING 1 -j CONNMARK –restore-mark
+ iptables -t mangle -I PREROUTING 2 -m state –state NEW -j NB_CT_PRE 2>/dev/null
+ iptables -t mangle -I PREROUTING 3 -m state –state NEW -j NetBalancer
+ iptables -t mangle -I INPUT 1 -m state –state NEW -j NB_CT_POST 2>/dev/null
+ iptables -t mangle -I OUTPUT 1 -j CONNMARK –restore-mark
+ iptables -t mangle -I OUTPUT 2 -m state –state NEW -j NB_FO_PRE
+ iptables -t mangle -I OUTPUT 3 -m state –state NEW -j NetBalancer
+ iptables -t mangle -I OUTPUT 4 -j OpenVPN
iptables -t mangle -I POSTROUTING 1 -m state –state NEW -j NB_CT_POST 2>/dev/null
iptables -t mangle -I POSTROUTING 2 -j NB_STAT 2>/dev/null
fi
$SCRIPTS/nb_vpn 2> /dev/null
$SCRIPTS/nb_setautomarking 2>/dev/null
+ echo 300 > /proc/sys/net/ipv4/route/gc_min_interval
+ echo 360 > /proc/sys/net/ipv4/route/gc_timeout

[atheling]

I probably made things more confusing that I should of. Each generation of the patch was intended to be applied against the beta 12 version of ZeroShell. So applying the patches successively will not work. Just apply the most recent patch against an unmodified version of the beta 12 code.

[Pit]

Z e r o S h e l l – Net Services 1.0.beta12 May 11, 2010 – 19:13

I have done so. May be you did not show me the latest patch.

[Pit]

Well, the patch is ok. Don’t know why the patch command failed.
I patched the rejected parts handish and saved the patched files
in /Database/custom.
Then under the “Startup/cron” tab under “SYSTEM” “Setup”, I have put the following into the “pre-boot” script:

for file in /Database/custom/*
do
cp ${file} /root/kerbynet.cgi/scripts/
done

All works fine with balancing disabled. Even vpn99 works well. All virtual servers behind ZS are reachable. No need for virtual servers to reach ZS from the outsite. No need for NAT on the only default gateway.

But the crux is: If i enable balancing the hole ZS is unreachable from the outsite and no virtual services are reachable.
Enabeling NAT on the balanced devices has no effect.

My point of view is outsite. I have to manage ZS and the servers behind ZS remote.
I tried ZS for balancing traffic to the outsite additionally.

Any idea?

[Pit]

Further testing:

The idea: creating a balancing rule that sends answer packages to a host through the gateway they come from.

It’s a funny thing. The log’s show that those packages are handled but only sometimes.

Bye the way the ZS experiment is very expensive meanwhile.

Is there somebody outsite who can configure the machine for us the right way?
Please make an offer.

Regards
Pit

[atheling]

Could you enable load balancing and then issue the following commands from the shell:

iptables -t mangle -L -vn
ip rule list
ip route list table 101
ip route list table 102
.
.
.
ip route list table main

(Where the 101, 102, … is for each of the tables that is shown in your “ip rule list” output.)

[Pit]

Meanwhile i found a working configuration with default gateway + one pppoe line
balanced.

iptables -t mangle -L -vn :

Chain PREROUTING (policy ACCEPT 697 packets, 119K bytes)
pkts bytes target prot opt in out source destination
789 135K CONNMARK all — * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
789 135K NetBalancer all — * * 0.0.0.0/0 0.0.0.0/0

Chain INPUT (policy ACCEPT 399 packets, 42348 bytes)
pkts bytes target prot opt in out source destination
489 57946 NetBalancer all — * * 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT 300 packets, 76696 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 507 packets, 125K bytes)
pkts bytes target prot opt in out source destination
507 125K NetBalancer all — * * 0.0.0.0/0 0.0.0.0/0
507 125K OpenVPN all — * * 0.0.0.0/0 0.0.0.0/0

Chain POSTROUTING (policy ACCEPT 807 packets, 201K bytes)
pkts bytes target prot opt in out source destination
115 8517 NB_CT_POST all — * * 0.0.0.0/0 0.0.0.0/0 state NEW
807 201K NB_STAT all — * * 0.0.0.0/0 0.0.0.0/0

Chain NB_CT_POST (1 references)
pkts bytes target prot opt in out source destination
30 2236 MARK all — * * 0.0.0.0/0 0.0.0.0/0 realm 0x66 MARK set 0x66
50 3875 MARK all — * * 0.0.0.0/0 0.0.0.0/0 realm 0x64 MARK set 0x64
115 8517 CONNMARK all — * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save

Chain NB_STAT (1 references)
pkts bytes target prot opt in out source destination
31 2312 all — * * 0.0.0.0/0 0.0.0.0/0 MARK match 0x66
51 3986 all — * * 0.0.0.0/0 0.0.0.0/0 MARK match 0x64

Chain NetBalancer (3 references)
pkts bytes target prot opt in out source destination
0 0 LOG all — ETH00 * 192.168.7.201 62.75.202.23 state NEW,RELATED,ESTABLISHED limit: avg 10/min burst 15 LOG flags 0 level 4 prefix `NetBalancer/001′
0 0 MARK all — ETH00 * 192.168.7.201 62.75.202.23 state NEW,RELATED,ESTABLISHED MARK set 0x64
182 31383 ACCEPT all — * * 0.0.0.0/0 0.0.0.0/0 MARK match !0x0
0 0 LOG tcp — ETH00 * 192.168.7.201 85.25.135.90 state RELATED,ESTABLISHED limit: avg 10/sec burst 15 LOG flags 0 level 4 prefix `NetBalancer/002′
0 0 MARK tcp — ETH00 * 192.168.7.201 85.25.135.90 state RELATED,ESTABLISHED MARK set 0x64
0 0 ACCEPT all — * * 0.0.0.0/0 0.0.0.0/0 MARK match !0x0
0 0 LOG all — ETH00 * 10.0.0.0/24 0.0.0.0/0 state NEW,RELATED,ESTABLISHED limit: avg 10/min burst 15 LOG flags 0 level 4 prefix `NetBalancer/003′
0 0 MARK all — ETH00 * 10.0.0.0/24 0.0.0.0/0 state NEW,RELATED,ESTABLISHED MARK set 0x66
0 0 ACCEPT all — * * 0.0.0.0/0 0.0.0.0/0 MARK match !0x0

Chain OpenVPN (1 references)
pkts bytes target prot opt in out source destination

root@zeroshell root> ip rule list
0: from all lookup local
32764: from all fwmark 0x66 lookup 102
32765: from all fwmark 0x64 lookup 100
32766: from all lookup main
32767: from all lookup default

root@zeroshell root> ip route list table 102
213.191.64.45 dev ppp1 proto kernel scope link src 78.51.19.196
213.191.64.52 dev ppp2 proto kernel scope link src 78.52.122.111
213.191.64.48 dev ppp0 proto kernel scope link src 78.51.125.38
87.234.250.0/29 dev ETH01 proto kernel scope link src 87.234.250.3
192.168.7.0/24 dev ETH00 proto kernel scope link src 192.168.7.75
192.168.250.0/24 dev VPN99 proto kernel scope link src 192.168.250.254
default dev ppp0 scope link
unreachable default metric 99

root@zeroshell root> ip route list table 100
213.191.64.45 dev ppp1 proto kernel scope link src 78.51.19.196
213.191.64.52 dev ppp2 proto kernel scope link src 78.52.122.111
213.191.64.48 dev ppp0 proto kernel scope link src 78.51.125.38
87.234.250.0/29 dev ETH01 proto kernel scope link src 87.234.250.3
192.168.7.0/24 dev ETH00 proto kernel scope link src 192.168.7.75
192.168.250.0/24 dev VPN99 proto kernel scope link src 192.168.250.254
default via 87.234.250.1 dev ETH01
unreachable default metric 99

[Pit]

Sorry, the main too.

root@zeroshell root> ip route list table main
213.191.64.45 dev ppp1 proto kernel scope link src 78.51.19.196
213.191.64.52 dev ppp2 proto kernel scope link src 78.52.122.111
213.191.64.48 dev ppp0 proto kernel scope link src 78.51.125.38
87.234.250.0/29 dev ETH01 proto kernel scope link src 87.234.250.3
192.168.7.0/24 dev ETH00 proto kernel scope link src 192.168.7.75
192.168.250.0/24 dev VPN99 proto kernel scope link src 192.168.250.254
default
nexthop via 87.234.250.1 dev ETH01 weight 1
nexthop dev ppp0 weight 1

[atheling]

Hmmm. Have you restarted/rebooted Zeroshell after applying the patch?

The reason I ask is the magic that gets incoming connections to have packets returned to the same gateway is in the NB_CT_PRE chain which the patch adds. However I don’t see that chain at all in your iptables listing.

There should be an entry that looks something like this:

Chain NB_CT_PRE (1 references)

pkts bytes target     prot opt in     out     source               destination

25951 1319K MARK       all  --  ppp0   *       0.0.0.0/0            0.0.0.0/0           MARK set 0x66

45921 2993K MARK       all  --  ETH01  *       0.0.0.0/0            75.144.252.181      MARK set 0x65

And the PREROUTING chain should look something like this:

Chain PREROUTING (policy ACCEPT 30M packets, 18G bytes)

pkts bytes target     prot opt in     out     source               destination

16M 9594M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK restore

517K   38M NB_CT_PRE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW

517K   38M NetBalancer  all  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW

Your PREROUTING chain does not have the reference to NB_CT_PRE nor does NB_CT_PRE exist.

[Pit]

Yes, i have rebooted.

The properly patched files are in /Database/custom. But for some reason the pre-boot-script does not copy the files.

for file in /Database/custom/*
do
cp ${file} /root/kerbynet.cgi/scripts/
done

Or are they overridden later with the original one’s?

Where can i find the original files in the /Database?
Perhaps it helps to copy the patched files there?

[atheling]

@Pit wrote:

Yes, i have rebooted.

The properly patched files are in /Database/custom. But for some reason the pre-boot-script does not copy the files.

for file in /Database/custom/*
do
cp ${file} /root/kerbynet.cgi/scripts/
done

So your “Pre Boot” window looks like this (ignoring the modprobe line)? And the preboot script is enabled (check box on upper right part of window)?

And your files are located in /Database/custom like this?

root@zeroshell custom> pwd

/Database/custom

root@zeroshell custom> ls -l

total 52

-rwxr-xr-x    1 root     root         5829 Apr 19 16:13 failoverd

-rwxr-xr-x    1 root     root         1926 Nov 30 19:51 fw_initrules

-rwxr-xr-x    1 root     root        13261 Nov 30 19:32 fw_makerule

-rwxr-xr-x    1 root     root         1147 Apr 19 15:51 fw_start

-rwxr-xr-x    1 root     root          199 Nov 30 11:30 fw_viewchain

-rwxr-xr-x    1 root     root         1996 Apr 19 15:54 nb_fw

-rwxr-xr-x    1 root     root         3392 Apr 16 11:43 nb_setautomarking

-rwxr-xr-x    1 root     root         1581 Apr 19 15:38 routeupd

-rwxr-xr-x    1 root     root         2335 Nov 24 20:36 setinterface

root@zeroshell custom>

@Pit wrote:

Or are they overridden later with the original one’s?

Where can i find the original files in the /Database?
Perhaps it helps to copy the patched files there?

I am not sure where the original ones are, I believe that they are decompressed out of a boot image but I haven’t investigated it. From an administrator’s point of view the only items that survive reboot are in /Database. But as you have found, the original scripts are not there.

[Pit]

Oh, oh alzheimer light. It is the second time i trapped with this window. In my firefox this window is to small. So i can not see the status checkbox.

No check -> no pre-boot-script -> no copied files -> ???

All works fine. It is a pleasure now to work with ZS. So lots of thank to all developers of this fine tool. Spezial thank to atheling for the fast and competent help.

I know that this is not self-evident and appreciate this.

Regards

Pit

[atheling]

I am glad that I could be some help.

If I recall correctly, Fulvio has indicated that the next beta for Zeroshell will be in June and that it should include my patches. So it should not be long before all these hassles will be a thing of the past.

[Author]

Posts