The former X Factor finalist had been linked to the new series earlier this year as rumours of who the judges would be circled around.
And Olly has confirmed that the rumours were true but he took the "massively tough" decision to say no to the show.
"It didn’t feel right at the moment," Olly said, "To go the whole circle with ‘The X Factor’ would have been amazing.
"I think I’d be really good for the contestants, too. But, I don’t want to distract from where I was musically”.
He added: “It was an appealing offer to sit next to Cheryl – I could have stopped her from marrying Jean-Bernard."
Olly admitted that doing Xtra Factor was tiring enough, confessing: “When I was presenting The Xtra Factor I was burnt out.
"I can only imagine what it would be like being a judge on X Factor and having a career at the same time.”
But the 2009 runner up - who releases a new album later this year - won't rule out joining the show in the future
He said: “I’ve always said I’d love to do it. It would depend on Simon and where the show goes."
The X Factor continues Saturday and Sunday nights on ITV.