"Everyone used to think that you liked being the token Black person in the room... I never did," the supermodel said