Show boss Simon Cowell has said the "everyone wants" the trio of newbies to be back for a second series.
The music mogul also says he no longer cares as much about viewing figures and claims that more young people are tuning in to the show.
"My perspective on life is slightly different than it used to be," Simon said this week. "I used to be really competitive, read the ratings and have white knuckles. Now I find it funny."
He told The Sun newspaper: “At the end of the day I always go back to Eric [Simon's son], who will hopefully one day be running these shows — one day, please God.
"But our ratings with young people are up and everyone wants Robbie Williams, Ayda Field and Louis Tomlinson to come back.”
He added to the Daily Mirror newspaper: “I’ve spoken to Robbie who with Ayda definitely wants to come back, I hope Louis does too. They have all had a great time.
“I’ve known Robbie a long time and it’s no secret we’ve been trying to get him to do the show for years. He says it’s the best job he has ever had because it is fun.”
However Robbie has joked he wants a pay rise for a second series, despite a rumoured £10 million fee as it is.
“We would love to come back. But we need to be treated kindly, so we’ll see. Is that my pitch for a pay rise? Abso-f***ing-lutely," said Robbie in an interview with The Sun.
He went on: "Honestly, The X Factor has been my favourite thing that I’ve done in showbiz and a welcome break from the thing that I’ve been doing for the last 30 years. I’ve always dreamed of having my own TV show. I want to do something like the Morecambe and Wise Show meets the Dean Martin Show meets Noel’s House Party.
“I’m in with all the TV bigwigs now. And they appear to be happy with me.”
It comes as Simon revealed that ITV have offered a new three-year-deal for both The X Factor and Britain's Got Talent.
Simon said: "We’ve just been offered an extension on both shows because of the 16 to 34-year-old audience. We could probably do three years or five years. It makes a lot of money. We got to a point years ago where we were able to compete with Strictly because there was that huge audience.
“We’re losing people now but fortunately the young audience has actually grown this year and all the advertisers and the sponsors want that income on a Saturday.”
ITV however has yet to officially announce a renewal, with both The X Factor and Britain's Got Talent running until the of 2019 as things stand.
For now, The X Factor concludes tonight at 7:30PM on ITV.