It's no real surprise, with claims that Tulisa wouldn't be returning for a third series as early as last December.
All but officially announcing Tulisa's axe from the show, Simon teased a new fourth judge for this year's panel.
He told the Radio Times this week that the new judge would be a gamble, saying that they would appeal to the middle aged and younger audience members.
Also in the interview, Simon ruled himself out as a judge on this year's show.
But he said he will be back in the future: "There is something about this show which is closest to my hearts, so at some point I will definitely come back on it"
It's currently unclear just who will join this new judge, with Louis Walsh claiming this week that no one has yet been signed.
He said: "We don't know officially, no one's been signed.
"There's rumours about Sharon Osbourne..."
Meanwhile, it was reported this morning that Nicole Scherzinger was torn over whether or not to return, with plans for a new album release in November.
Auditions for the X factor 2013 started in London on Saturday, and continue in Birmingham, Cardiff, Manchester and Glasgow this month. Judges auditions will take place later over summer.
Browse our gallery from the first round of open XFactor auditions in London at the weekend below...