NYSE Euronext (NYX), the biggest U.S. exchange operator, will pay $5 million to resolve regulatory claims that the New York Stock Exchange violated rules by giving certain customers a head start on trading information.