The ironic thing is that we had the means whilst in the EU to enforce stricter regulations at the border, and could've easily done far more to close down agencies that were actively searching overseas for cheaper labour. If you're in a working-class area and a Labour voter, logic would dictate you'd have solidarity with the rest of the working-class regardless of origin, and your main gripe would *overwhelmingly* be with Westminster, who've long abandoned the North, opting to push public and foreign direct investment into the City of London and its surrounding areas.
We'll come to see that Brexit will ultimately bring more immigration - when needing to tie up trade deals with developing economies such as India, they'll ask for three things: visas, visas and more visas - and they'll get them. That's not to say this is a bad thing; given the ever-worsening dependency ratio in the UK, we'll be actively courting immigration for the next 30 years at least so as to continue the provision of an adequate pension system for a new generation of elderly people.