That's all true and in reality I'm far from advocating removing it, however I do have issues with the blotting out of aspects of history and the legacy they leave.The difference I personally have with the Colosseum is that it is genuinely a really good mini museum, with lots of literature on site about its history. It is truly an engineering marvel as well.
However lots of statues have so such educational material around then and are often pretty gross to look at to boot.
Where does it stop? James Penny, a slave-ship owner, had a street named after him in Liverpool, which some band then used as a name for a track of theirs.
History is often signified by how relevant it is in the human consciousness or today's zeitgiest; this is why places like Auschwitz are maintained or other memorials.
If you travel around Eastern Europe, there's multiple relics of a system that is now defunct and quite often despised, but they're left as a mark to remember.
Most are good but some are bad, but they're a part of the fabric of history. For me, the statue isn't a celebration of slavery but an artefact or source linked to it.
@Tubey mentioned it earlier that people now look back on history and believe that our morals inherently match those of the time, when in reality they do not.