Four cast members were let go in the wake of past racially insensitive behavior and social media posts that had resurfaced