If you apply Atheling’s patch for mangling then you can use both QoS and Netbalancer. I don’t understand what do you mean exactly
everytime I switch between the interfaces
could you elaborate a bit? Failover sticks to the active connection and switches to the standby only when the active goes off.