Former X Factor finalist Olly replaced Dermot O'Leary on the show in 2015, co-hosting alongside Caroline Flack.
The switch up lasted just the year, with Dermot back in 2016 after his replacements were panned by viewers.
Now reflecting on his time fronting the show that made him famous, Olly admitted: "The X Factor wasn't a nice experience. I was targeted by trolls and the media. It took me a few years to pluck up the courage to speak to someone."
He told OK! magazine this week he received therapy to help him move on.
"I only went a couple of times but talking to someone who wasn't judgmental helped me to understand why I was reacting or feeling the way I was," he added.
Olly recently admitted he turned down the chance to return to The X Factor this year as a judge.
The 'Moves' singer said no, instead deciding to stay as a coach on rival singing show The Voice UK.
"I have so much respect for Simon and it was difficult to tell him No – I’m not sure we’re on talking terms," Olly said. “I felt I had such a great time on The Voice, why would I leave after one year?
"I felt I did an incredible job last year and I’m very proud of that.”
Meanwhile, Olly will be back on TV this Christmas with his very own TV show.
ITV teased: "The much-loved pop star and coach on The Voice UK will be heading back to his Essex roots as he takes over an entire pub to put on a jam-packed night of games, chat and performances of all his biggest hits."
Bradley Walsh, Emma Willis, Sir Tom Jones and Jennifer Hudson and more will feature in the programme.