The number of migrants coming to Britain must be curbed to slow population growth, Immigration Minister Phil Woolas has warned.