ITV confirmed earlier this month that alongside Simon, three new judges will join this year's series while Sharon Osbourne will act as a fifth judge for the live shows.
Both Nicole Scherzinger and Louis Walsh have quit the series, the channel confirmed.
Show bosses have described the change as 'the start of a new era' for the show, which will air its 15th series this year.
As yet, the names of the new judges have not been officially confirmed but rumour has it Robbie Williams and his wife Adya could be involved.
Now Simon has suggested that is indeed the case in an interview with the Daily Mirror newspaper.
“It all kind of worked out without me even asking him," Simon is quoted as telling the tabloid. “He came down to Britain’s Got Talent, had such a good time that night and goes, ‘you know what, this is fun’.
“He comes back to my house... ‘Can we do a deal?’ We do a deal. I think it’s the best thing. And you know, because she (Ayda) is a member of the public, that makes it better.”
Appearing on ITV's This Morning last month, Robbie himself certainly didn't deny the rumours.
Although he also insisted that (at least as yet) he hadn't signed any contracts.
Robbie told hosts Holly Willoughby and Phillip Schofield: "If there is an X Factor in my future whether it be this year or next year or never, I am scared for my own safety and health."
He explained: "Not because of the people out there or the public or Twitter or Instagram, but my wife.
"She's a sort of Grade A+ student from Beverly Hills, incredibly square and is like, 'Why would you do that?'"
Robbie went on: "There's a lot of me that would love to do it too but there's nothing there.
"Me and Simon and Lauren and the kids and little Eric hang out an awful lot and talk about a lot of things.
"Nothing's signed but watch this space and I guarantee you will be the first people I speak to about it."
The X Factor 2018 will air on ITV later this year.