Please post your QOS rules. Without seeing them it is impossible to guess what’s wrong.
I had similar experience when messing up with QOS for VOIP – one mistake and everything grinds to a halt.
But when I got my rules rigtht, VOIP quality actually improved somewhat.