The former finalist, who was runner up in the 2009 series, says he instead decided to stay on rival singing show The Voice UK for a second series.
Ahead of The Voice's return this weekend, Olly confirmed Simon Cowell asked him to return to The X Factor last year.
Olly told Good Morning Britain: "He did in the summer. Listen, I love Simon, love X Factor, nothing but love for the show.
"But just being on The Voice with the coaches, it wasn't about money. It was more about the chemistry we had as a team. If one of them wasn't coming back it might have been a different decision.
"I just love the show, I love the people on it and they gave me an amazing opportunity as a coach and to help other people. So I'm having a great time on it."
In a separate interview with the Daily Mirror, Olly added: "I have so much respect for Simon and it was difficult to tell him No – I’m not sure we’re on talking terms.
“I felt I had such a great time on The Voice, why would I leave after one year?
"I felt I did an incredible job last year and I’m very proud of that.”
Olly meanwhile told the BBC that he and Simon remained "good friends".
He said: "We've got nothing but respect for each other. And I've got nothing but respect for what he's given me in my career."
Olly returns to The Voice UK in 2019 alongside fellow coaches Jennifer Hudson, Tom Jones and Will.i.am.
Olly raved: “Now I’ve got one season under my belt, I can’t wait to see what my second season has in store for me. Last year I managed to get seduced and picked some entertaining performers. The other coaches might think they have worked me out but I’m coming back with a bang, just you wait!”
The Voice UK 2019 begins its blind auditions this Saturday night at 8PM.