"[Meghan] knows the trip to England has been difficult for Harry. He didn't want to leave Meghan and Archie alone," the source tells PEOPLE
