They thought Brexit could protect them from new waves of immigration, but now, want to change their minds.