Despite currently sitting in The Voice UK's spinning red chairs, Olly thinks it'd be cool to end up where it all started.
Olly rose to fame on The X Factor back in 2009, where he finished runner up to winner Joe McElderry.
Currently Olly is on The Voice UK alongside Will.i.am, Sir Tom Jones and Jennifer Hudson.
But ahead of its final this Saturday night, he told Closer magazine: “It would be amazing to judge alongside Simon.
“I have so much respect for him, and it would be crazy for my career to at one point be sat next to him.”
As it happens, there may well be space on The X Factor panel this year.
Olly's comments come amid rumours of a big shake up to the 2018 judging line up.
It could see Nicole Scherzinger, Louis Walsh and Sharon Osbourne ALL leaving.
The dramatic move may be made despite Louis and Sharon already signing contracts for 2018.
"Louis and Sharon are both contracted to return this year, but Simon and the producers have lost faith," an insider alleged to The Sun newspaper previously.
Meanwhile, Olly has dismissed rumours of a feud between himself and X Factor boss Simon Cowell.
He said: “I saw him last year at an ITV event and he was fine. Listen, there’s no beef between myself and Simon.
“We’ve got a great relationship and I’m sure when I see him next we’ll have a drink and we’ll be fine.”
The Voice UK 2018 final airs this Saturday night at 8:30PM on ITV.