ITV confirmed this month that it had agreed a new deal that will see the show, and its sister programme Britain's Got Talent, airing through until at least 2016.
But with Gary Barlow and Sharon Osbourne both set to quit at the end of the current series, bosses are already making plans for another panel revamp.
According to The Sun, X Factor boss Simon Cowell is keen to get Rita Ora on board, with the popstar having previously appeared as a guest judge in 2012.
It was then claimed that Rita had been offered a full time spot on the panel but turned it down, with Rita quoted as saying: “I love it but if I was to take on a full-time role it would be when I’m loads of albums in.”
In the end, Nicole Scherzinger took up the free spot on the panel after previously appearing as a guest judge as well.
But now she could be on the way out, with Cheryl Cole also tipped to join next year's judging line up.
"Rita is at the top of his list with Cheryl. He has been a massive fan of hers for a while," a source said today. “He has offered it to her before but she didn’t think the time was right. Now the ball is very much in her court.
“Simon is planning a huge shake-up next year, as it’s highly likely he’ll be back. Rita and Cheryl would be his dream team.”
It was claimed last week that Nicole had been put out by all of the speculation over who could be joining the show next summer.
A friend of Nicole's told heat magazine: "It's a real slap in the face.
"She puts everything into her job and when she heard [about Cheryl] she was furious."
The X Factor 2013 live shows continue next Saturday and Sunday with a Best of X Factor themed week.