The fact that for many years, even after the A8 accession, immigration from non-EU countries was comparable to that from EU countries suggests that things weren't that unfair, especially as on most objective criteria non-EU migrants were less educated, would be less likely to be employed, and so on. Again, they're the ones that are "managed", so it's a funny world where you would take what appear to be poorer outcomes and say we should do more of that and less of the butting out, even when that produces better outcomes. We're also omitting, of course, the removal of our right to live and work in 27 countries, which always seems to be ignored as though that's a piddling thing to have stripped from us.
For what it's worth, I don't believe there will ever be open borders because it's too easy for politicians (whether right wing ones scaring us about brown people sullying our racial purity or left wing ones scaring us about migrants undercutting wages) to trot out lies, and for their supporters in the media to amplify those lies, against a part of society that has no vote and much less power than native citizens. Few people actually have the time, nor I suspect the inclination, to actually bother researching things, so they swallow whatever fits their heuristic of the world, and there's a strong evolutionary tendency to fear the out-group. If you're happy to go along with that charade then by all means, that is your prerogative, but we'll all be worse off for it.