Youth and Sports Minister Khairy Jamaluddin says people shouldn't be speculating about a possible cabinet reshuffle because it is up to Prime Minister Najib Abdul Razak.