One way of seeing it - for me I see the "privilege" as unnecessary. For any "loss" the net result of equality and the knock on effect of socio-economic rebalance for everyone would counter it. The only loss would be for the truly privileged in the long run, which they can afford to take.
I don't think of racial equality solely in terms of race, so I think that's why I have contrary views on it. For me, class and economic subjugation is a far bigger issue and one that creates the climate for racial discrimination to exist. In my eyes, lower class whites are racist solely due to grievances seen in their own lives and looking for a scapegoat.