The singer was always a strange choice for the main show given his lack of presenting experience, and at times it definitely showed.
But Olly reckons he did well in the circumstances and says he "loved" the opportunity.
The former X Factor finalist told the Metro newspaper this week: "People seemed to forget the limited amount of experience I'd had with live TV and how difficult it is.
"When you see presenters who've done loads of TV like Ant and Dec and Dermot O'Leary, they can make it look easy because they've done it so long."
Olly added: "In my heart, I felt I did a good job and there were a few hiccups along the way, but that's what live TV is all about."
It was announced last month that Olly and his X Factor co host Caroline Flack wouldn't be back, with Dermot O'Leary back in charge for this year's series.
Now Olly is looking forward to put 100% into his new album, due out later this year.
"[Music] has been at the forefront of everything that I have been doing," he told the newspaper. "I wouldn't have got the X Factor job if it hadn't been for the success I've had with music.
"To do my new album and promote it as well as doing The X Factor would have been too difficult."
Olly went on to comment: "I made the decision to leave. It's a great show.
"I loved doing it. It was a boot camp for me from star to finish, but a great experience. I'm looking forward to finishing my new album and releasing it."
The X Factor 2016 will air on ITV later this year.