Four men have been arrested on suspicion of aggravated trespass at the Prime Minister's constituency home in Kirby Sigston, North Yorkshire Police said.