The Leader of the Opposition lost his Stone House ward seat as Conservatives retained control of the Borough Council.