PM can’t ignore damage inflicted by minister’s taste for high life