Ex-Liberal Party powerbroker says ‘the party is over’