Actor Michael B. Jordan has joked he'll "let it slide" after Paul Rudd replaced him as PEOPLE magazine's Sexiest Man Alive.