Mate, you said...
"Not everyone who voted to remain in the EU are racist, but all racists voted to remain in the EU."
'Makes just as much sense' as saying.
"Not everyone who voted to leave the EU are racist, but all racists voted to leave the EU."
I've just given you two examples of why "all racists" didn't vote to remain in the EU.
So you assertion that your statement makes "just as much sense" as the other is invalid, because one is possible and the other is impossible, because there's no notable example of a notable racist group of people wanting to remain in the EU.
This is simple logic.