The former presenter of the show will follow fellow X Factor alumni James Arthur and Little Mix and perform his new single.
Olly had hosted last year's series alongside Caroline Flack before Dermot O'Leary returned to replace them this year.
In an interview with the Daily Mirror newspaper, Olly claimed he was happy to have returned to music.
Although he still reckons his presenting wasn't THAT bad.
"I feel that we got the hard end of the brush really. I don’t think we did a terrible job. We all make mistakes," Olly said.
He added: "There’s no hard feelings, but I’m so happy to be back to my music."
Olly insisted that despite the axe, his relationship with Simon Cowell and the show was "still amazing".
"That’s why I’ll be back in a few weeks to perform music,” he said.
Olly's latest single You Don't Know Love made it to number 15 in the charts earlier this year, ahead of his fifth studio album 24 HRS which is out in November.
Alongside James Arthur and Little Mix, Olly will join OneRepublic, Shawn Mendes and John Legend as this year's guest performers so far.
The X Factor 2016 is back Sunday night at 8PM on ITV on Sunday night with the latest results.