Queen Elizabeth was "very hurt" when Prince Harry and Meghan, Duchess of Sussex stepped down from royal duties and left the country.