Russell Brand finds it hard to "let go" of what other people think of him, because he's always struggled with fame.