Albanese has lost face and given ground. Ley should follow suit