The former finalist took over from Dermot O'Leary alongside Caroline Flack on last year's series but the pair had a tough time of it.
Reflecting on the experience, Olly said this week: "The X Factor was amazing but the schedule was hard. Then to add music into that makes it doubly hard."
He told The sun newspaper: "I'm a human being, I can't do everything. People forget you are on the biggest show on TV so as soon as you walk out on Saturday or Sunday night all the attention is on you."
Olly confessed that he "loved the attention" however found every night "just so intense."
He explained: "It's the mental pressure of dealing with TV, the lights being on you, plus at the same time promoting your single and wanting that to do well."
As the series went on, Olly admitted it all got a bit too much.
"In the lead-up to Christmas it hit me hard. I had a week in bed when I couldn't move," he told the tabloid. "It was a bit of a brain freeze. I was a bit of a zombie. I had to have three or four days in silence when I just chilled out, got off my phone and had a bit of me time."
Olly and Caroline stepped down from the show earlier this year, with Dermot returning to take over for the new series.
The X Factor will be back on ITV later in the autumn.