Four elderly residents and their caregivers were robbed during a home-invasion robbery in Southern California late Thursday, police said.