The former finalist, who was runner up in the 2009 series, says he instead decided to stay on rival singing show The Voice UK for a second series.
Olly released his sixth studio album You Know I Know on November 9, led by new song Moves, which he performed on The X Factor last weekend.
And ahead of his record hitting shelves, Olly told The Sun newspaper: "I was nearly on the X Factor panel but I decided to stay at The Voice.
“It was amazing to be asked, but The Voice is more for me at the moment and my career. Who knows about the future but I wanted to stay on The Voice for now.”
In a separate interview with the Daily Mirror, Olly added: "I have so much respect for Simon and it was difficult to tell him No – I’m not sure we’re on talking terms.
“I felt I had such a great time on The Voice, why would I leave after one year?
"I felt I did an incredible job last year and I’m very proud of that.”
Olly meanwhile told the BBC that he and Simon remained "good friends".
He said: "We've got nothing but respect for each other. And I've got nothing but respect for what he's given me in my career."
It was confirmed earlier this year that Olly would return to The Voice UK in 2019 alongside fellow coaches Jennifer Hudson, Tom Jones and Will.i.am.
Olly raved: “Now I’ve got one season under my belt, I can’t wait to see what my second season has in store for me. Last year I managed to get seduced and picked some entertaining performers. The other coaches might think they have worked me out but I’m coming back with a bang, just you wait!”
The Voice UK will air on ITV in the new year.
For now, The X Factor live shows continue Saturday and Sunday nights on ITV.