I do feel for the non racist leave voters because it must be bad to be always deemed a racist because the reasons you voted leave is that you didn't want foreigners in the country.
Joking aside, I think leave voters who are not racist need to live with the fact that some (a lot) of leave voters are racist in various guises, whether that is casual or full blown neo-nazi. This is a fact and the racist card is going to be brought up, as they say 'you have made your bed lie in it'
It betrays two things really. Firstly there is a lack of comprehension about the lump of labour fallacy. I mean in 1800 the population of the UK was around 10 million (according to the first census so we have to imagine that millions of Europeans didn't fill it in, but still...), and yet despite our current population being around 68 million, we still seem capable of finding jobs and prospering as a nation. Indeed, one would think the fact that the population of the aforementioned Nelson has fallen by over 25% in the last 100 years might give some folk a clue that a growing population isn't actually a bad thing (quite the contrary, as the fortunes of Nelson illustrate).
Secondly, there seems to be a willingness to impose restrictions on the movement of non-British people that they wouldn't dream of imposing on British people. I mean it would be regarded as ludicrous to prevent someone moving from Birmingham to Manchester because they're not qualified enough or the job they'll be doing isn't on an approved government list. I dare say we'd even find such impositions absurd if we applied it to movement between the four countries of the United Kingdom. And yet we're happy to impose that on foreign people because, erm, well, I'm not really sure why, but as I've illustrated dozens of times, the legal definition of racism does include treating people differently on the basis of their nationality as well as their religion, race, and so on.