Prime Minister Tan Sri Muhyiddin Yassin says some quarters are opposing him because he refused to give in to their demands, including to drop criminal charges against certain individuals.