It’s not bigotry to worry about migration – the latest figures tell a complex story