"It seemed like this really unfortunate thing at a really bad time," the Double Agents competitor tells PEOPLE of her surprising exit
