The Prime Minister has called on members of his party to pull together and encourage more of the public to back them.