The International Monetary Fund on Friday warned that Britain's potential exit from the European Union posed a "significant downside risk" to the economy.