Yes, the problem is men like Gregg Wallace – but it’s also those who should stop them and don’t