A Kent conservative councillor says he feels let down by the Health secretary after photos emerged of an alleged affair with one of his aides in the Sun newspaper.