But he has come under fire from the county's only Labour MP who said she had no idea he was paying a visit