Young voters and women have left the Coalition – and they aren’t being tempted back