Alison Rowat: It's not just Jeremy Clarkson who is being driven to despair by Labour