That is wrong my friend.
I know we live in an age where we can be whatever we pretend to be and can flitter between a male one day and a female the next but lets not be silly in claiming there is some unknown reason as to why all the statistics since the human race began show that males to be more agressive/dominant of the sexes.
Throughout human history - no matter what culture, skin colour, country men have constructed patricharal societies and its not a coincidence.
1. Physical strength
2. Inherent desire to be dominant
3. Testosterone and the aggressive tendency to be willing to fight and conquer.
That is at its core why and it feeds into wider society even today. If you walk through town on a Saturday night I would bet my life savings on that you would feel different in terms of your sercurity and apprehension levels to walking past a gang of drunk girls than through a gang of drunk young lads.
There is a reason why your senses become heightened when walking past the group of lads and its because males are inherently more dangerous and pose a bigger threat to you.
Statistics show it - more dangerous to others (see murder, rape, assault rates) and more dangerous to themselves (see suicide/addiction rates).
Going back to the asylum seeking topic - women and children are viewed by the "natives" differently to that of men. One are seen as harmless and simply being providers for their children, the other are viewed with suspicion by the natives and that feeds into our evolutionary history where tribes from other lands would rape and pillage nearby villages.
Instead of calling these people "racist gammons" you need to understand the reasons at the very basic core as to the mistrust and then you can educate people as not all of what you see is down to Right Wing nutters reading the daily mail.
This runs much deeper in human psychology.