Andy Cohen didn’t name names, but reveals that none of the “annoying” blacklisted guests included Bravo stars