The Duke of Sussex added that while his family relocated to California, he "can't get out" of the royal family
