Olly found himself under attack during the live shows of last year's series after taking over hosting duties from Dermot O'Leary alongside Caroline Flack.
The complaints hit a peak after Olly mistakenly announced Monica Michael had been sent home during one live results show, but he thinks the uproar was ridiculous.
Olly said (via The Irish Independent): “At the time the mistake happened there was such a massive tragedy and people were getting killed and all these things were happening around the world and all people wanted to talk about was my mistake on the X Factor and how I was going to get sacked.”
The former X Factor finalist previously revealed how the hate had left him stressed and depressed but in a chat with the Daily Star, says he'd still return to the show.
Just not as a host.
"I would do X Factor again but as a guest judge or judges’ houses as this is Dermot’s show," Oly explained. "It’s like Top Gear, Chris Evans is an amazing presenter but he was filling Jeremy’s shoes, it’s his show.”