John Legend was "traumatised" by the death of his grandmother because it led to his mother abandoning the family.