Yup, guess I wasn't using enough force
And I can confirm this is where a lot of rattles are coming from
I put some electrical tape between the plastic bits and the metal and behold, 80% less rattles
Just an fyi for anyone with a subwoofer who doesn't want to pay $ 300 for an audio company to scam you