Cheryl Cole is also in talks with producers to return although apparently the Girls Aloud singer has some reservations about agreeing to a deal just yet.
It's the fourth spot on the line that appears to be a mystery with various rumours linking stars to the show.
While Nicole Scherzinger had been tipped to return for her third series, now she's only seen as a "back up" choice after indicating she wanted to focus on her music career.
Instead, The Sun newspaper reports, bosses are hoping to get former X Factor guest judge Rita Ora on board.
The singer had appeared in the 2012 series and was originally offered a permanent spot ahead of Nicole.
But she said at the time: "I think right now I don’t think it would be the right time to do anything X Factor.
“I’d love to come back in and out and be a guest judge but if I was to take on the full role it would come from a place where I feel I’m satisfied with my career and right now I’ve got a lot to do.”
And if all else fails there's Dannii Minogue.
Speaking yesterday, X Factor boss Simon Cowell said: "Despite all the rumours, ITV and the producers have not decided on the rest of the judging panel quite yet. You’ll just have to wait and see..."