I noticed that the system load was around 17% when I was downloading which was way to high given the spec of the machine.
Because I had been changing lots of settings when trying to get the box setup in the first place I figured I could have something set incorrectly.
I thought it would be easier to just start again on the client machine. I got a new install configured in about 20 minutes and this one has been totally fine, I’m impressed with the link aggregation, seems to work fine now.
Incidentally, I have not enabled the net balancer, just make sure you use static routes to send your VPNs out over each individual internet connection you have connected.