I think centre back, more than any other position than goal keeper, is as much about judgement and ability to read the game as it is natural ability.
Carragher is the most obvious example. When he first broke through, I didn't give him a chance of a career lasting at Liverpool, and I didn't think that was bias either. But he stuck at it, as he grew he read the game well and whatever deficiencies he had with pace or ability on the ball, he covered for very well.
You could say the same about Terry, when he first broke through, I expected his Chelsea career to follow a similar path to Jody Morris, I thought he had too many limitations to make it as a top centre back.
And the other side of that argument, on paper Titus Bramble had everything you'd want in a centre half and everyone knows what his career has been like.
I'm not saying I'm sure Duffy will make it, but he has enough decent attributes that there is potential there.