Okay, back in the days of slavery and Jim Crow down in the southern United States, your average African-American was liable to be strung up for looking at or talking to a white lady. A whole mythology grew up around this of black men as "beasts" unable to control their primitive sexual urges, the fact that they were traded as animals with some slavers even going so far as to "breed" particular slaves led to a whole load of nasty imagery around black men as animals.
In the weird way repression works, these folks were worried that their dainty women folk would, once ravished by these sub-humans, enjoy it and turn their backs on the white race, this was the justification for the policy of lynching those seen in conversation with a white woman. Thus, in these people's warped subconscious they were not only beasts unable to control themselves but irresistible as well... And as a crude example of their heightened masculinity, the myth of the large penis was born.
David Remnick has a very good chapter about this in his book about Mohammed Ali.