"It's a little bit hard to say, 'I'm not racist,' when you get other reports from credible people," Sherri Shepherd said