Former doctor Robert Xie has lost an appeal against his conviction for murdering five members of his extended family in their Sydney home.