New 'X Factor' judge Louis Tomlinson has admitted Simon Cowell has annoyed him during his first series on the panel of the TV talent show.
