French Sports Minister believes border restrictions may limit the presence of international spectators