The former contestant and one time host of the show is currently a coach on rival series The Voice UK.
But it's been said Olly could well switch shows - depending on one thing.
The Daily Mirror reports that Olly's move to The X Factor depends on whether or not Robbie Williams accepts an offer.
The tabloid says that Robbie is the first choice for the job.
“We’re all huge fans of Olly’s, but Robbie is one of the biggest stars in the world," a source spilled, "When the opportunity came for [Robbie] to join, we couldn’t pass it up.”
Although it's claimed that Robbie has signed up, leaving Olly to stay on The Voice, Robbie himself has insisted nothing is official yet.
"There's a lot of me that would love to do [X Factor] but there's nothing there," he said this week on ITV's This Morning. "Me and Simon and Lauren and the kids and little Eric hang out an awful lot and talk about a lot of things.
"Nothing's signed but watch this space and I guarantee you will be the first people I speak to about it."
Meanwhile, Olly previously said he'd love to join The X Factor.
Speaking in April, he told Closer magazine: “It would be amazing to judge alongside Simon.
“I have so much respect for him, and it would be crazy for my career to at one point be sat next to him.”
The X Factor 2018 will air on ITV later this year.