Are you sure you'd take that position? Affirmative action in this country has disproportionally benefitted one group (white women) and the incarceration rate of poor whites isn't anywhere near the level of rates of poor people of color. That is a massive determining factor in the upward mobility of poor people.
Not at all saying poor white people are loaded with opportunity in the US, mind you.
Crime rates are something like 75% higher in urban areas than in rural areas, and poor people of color disproportionately live in urban areas while poor whites disproportionately live in rural areas, so I would say that what you describe is largely what we would expect from a legal system whose biases are more monetary-based than color-based. If those incarceration rates were equivalent, the conclusion that the legal system was biased against whites would be difficult to escape.
Arrest rate stats unambiguously suggest that the
police are biased when it comes to African-Americans specifically, which gets back to both the hostile occupying force thesis and systematic racism in wealthier areas. This then leads to crimes perpetrated by non-African-Americans against African-Americans tending not to be reported due to (rightful) distrust of the police, which also helps explain the disparity in incarceration rates.
Statistics are useful, but they don't work in isolation. It's important to understand how all the forces in play interact. If there's a residual that can't be explained by other statistics, that's when we have identified the effects of racism. For instance, the disparity in the "incarceration penalty" with respect to unemployment is at least partly explained by the observation that employers are less likely to call people with similar experience in their applicant pool with obviously ethnic names than those without. There are problems with the peremptory strike system that involve demographics, monetary factors and overtly racial issues.
As one climbs the economic scale out of poverty, I think that the factors in play quickly stack up to leave someone of color worse off. I don't think it's nearly as clear-cut once we start talking about poverty, though.