I was looking at this, the problem is the current implementation.
Instead of having a general mode switch Fail-over / Load balancing we should have 2 pools, one for Load balancing and one for fail-over.
All interfaces registered in Load balancing pool will act as a fail-over but the ones in fail-over will be enabled only after everything else fails. Weight could act as priority on fail-over. This would make sense especially if traffic on one interface is expensive.
Unfortunately I do not use a setup this complicated, I have only a fail-over on LTE.