More than 40 housing and community organisations have written to the Prime Minister, to express concern that migrant communities are being blamed for the housing crisis.