I agree that it is a bit tenuous calling him a racist to us Brits. But that doesn't mean his comments weren't bigoted and downright insulting to all Brits, whether they supported Brexit or not. And not at all appropriate comments from a member of the House of Lords.
His comments you quoted regarding immigration are so ridiculous they would be almost comical if they weren't so damaging. To say Cameron could, if he wanted, have controlled immigration by stopping all immigration from outside of the UK, is like saying you can control world population by nuking India and China. True but totally impractical. They were also arguably racist to people outside the EU.
The fact is, the uncontrolled immigration from EU is already impacting on the number of qualified people coming in from outside the EU, and it is impacting many area's of our economy and infrastructure.
Also, if there were no immigration from outside EU where would that leave seekers we currently help.