Well… I haven’t tried this particular scenario, so I can not claim to know it works, but essentially, you are looking for a complex solution that requires a bit of rethink on how you are allocating you ‘connections’.
Playing with the QoS in routed mode on my implementation, it seems the QoS is fairly simple to implement – as long as you aggregate to a connection to apply it to. It is simple and limited in it’s inteligence – but that makes it work better.
This is going to probably cost some processing power and add a bit of latency, but essentially, I think you need to configure a couple of additional interfaces:
Add BR4 and BR5 as the next step in the communications:
BR4 would have all traffic forwarded from BR0 and BR1
BR5 woudl have all traffic forwarded from BR2 and BR3
Apply QoS rules to the BR4 / BR5 connections.
It is somewhat nasty in the implementation, but it should solve the problem.