Just some info, I discovered how I ended up with all those entries: http://forums.gentoo.org/viewtopic-t-463726.html
I’ve actually managed to more or less force expire the conntrack count down to 10000 now (down from ~55000) and it freed about 65M of ram in the process.
I have this in my boot script now.
echo 3600 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
echo 16000 > /proc/sys/net/ipv4/netfilter/ip_conntrack_max (I know this is low but its restricted to 4096 by forces out of my control anyways)
echo 0 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_loose