The 'Troublemaker' singer hinted earlier this year that he wouldn't be returning to host the ITV2 series alongside Caroline Flack.
Confirming his decision today, Olly said: "It’s going to be a shame not to do The Xtra Factor again. I’m gutted because it’s been a huge part of my life for the last two years.
“But I think whoever Caroline decides to pick, whoever the bosses decide to pick to replace me is going to be great. I don’t know if she would do it on her own.
“And we’ll see what happens in the future. I mean, you never know."
Speaking to The Sun newspaper, Olly said he wanted to have a bit of time off before travelling to the states to help crack the American market.
“I remember watching the Big Reunion recently and one of the popstars was saying that you do sometimes feel more of a product than a human being," he explained. “I don’t get four weeks holiday a year — I only get one week or two weeks sometimes — and I don’t get weekends off like everyone else. I have missed that.
“Fundamentally, you need time off to refresh your memory and brain.
“I love it, but it is a lot of pressure.”
And Olly also suggested that moving away from The X Factor would be good for his music career too.
“There’s always this stigma with X Factor — people think you come off the show and you record in booths and you don’t even hear the songs, you just do them," he told the tabloid. “But I do actually write. I’ve had four No1 records so far and I’ve been a part of all four of those songs. I love the process of songwriting."