Robert Downey Jr. Is opening up about his time in prison in the late 90's saying it was "the worst thing that happened" to him. In 1999, Downey served a year in a California State Prison, after missing a number of court-ordered drug tests tied to a 1996 cocaine possession charge.