The singer, who was runner up on the show in 2009, says that hopefuls in recent series feel they've "already made it" after appearing on the show.
Olly explained: "I felt that over the years many contestants started believing their own hype and not necessarily focusing on what they were to do – be a successful musician.
"Simon (Cowell, X Factor judge) said the same on many occasions."
"That's not the truth – it's about what you do after the show that counts," Olly said.
Chatting to The Argus newspaper in Brighton, Olly also spoke about his brief stint as host of the main show last year.
Olly fronted the previous series with Caroline Flack but neither returned for this year's show.
But it sounds as though Olly would've turned down the chance even if he was asked.
The Troublemaker performer said: "Getting asked to host the biggest show on television is an opportunity you can't refuse.
"At the time I thought I'd like to branch out and go into TV, and that was great, but I was reminded that music is the most important thing to me."