"They both told production they weren't returning to the show months ago," an insider told PEOPLE of the Southern Charm stars
