Close to 100 Latino workers walked out from a construction site after a group was accused by their employer.