There is a ban on men telling women what to wear. It's enshrined in our civil rights, which states clearly that one person cannot control another, specifically over the age of 18. There's special rules in place for matters like school uniform and the right to marriage for minors; again, put in place to prevent bad things happening to young people.
Adults in our society have the right to do what they like within the law with an independent will.
In fact, this is actually upholding that civil right. It's making sure that Muslim men cannot dictate what a Muslim woman can and cannot wear. As I said, there's no reliable way of knowing if a woman is being forced into wearing clothing, so much in the same way that we ban slavery because there's no way of knowing if someone is voluntarily being a slave or not, so the banning of the burka is due to this.