Olly rose to fame on The X Factor in 2009 where he finished runner up.
The singer, who is now a coach on The Voice UK, thinks that the ITV series could benefit from taking some time out.
Speaking at the launch of The Voice 2020 about The X Factor, Olly told The Sun newspaper: "I'm still on the fence on the pure basis I love the show because I came from it, so of course I want to watch it on TV.
"But at the same time, maybe the show does need a little break just for people to replenish their eyes and ears and refocus.
"The show can come back with a massive bang in a couple of years time, but that's not up to me - it's up to Simon and the X Factor to make that decision."
Olly, who hosted The X Factor in 2015, added: "It's still a show I hold dear to my heart because it made me who I am."
The Voice UK returns this Saturday with Olly joined by new coach Meghan Trainor alongside returning panellists Sir Tom Jones and will.i.am.
Revealing his advice to the newcomer, Olly said: "I just told her to be herself. You trust your instinct, don’t be swayed by what will or Tom might be saying. They’re the experienced ones.
"For me it’s about following your gut instinct. That’s what I did last year with Molly and lots of other acts in the past.
"It’s what YOU believe in and what YOU want in your team, not what other people want you to pick so just follow your gut and keep those ears open."
He added: "I do think Meghan is my biggest competition. She’s got that fresh energy, she’s really determined to win. In my first year I had some really good contestants and I felt confident but in the end we didn’t get over that final hurdle and I think that was due to experience.
"Whereas this year, I think she’s going to be the one to watch. She’s got a great personality and she’s managed to get on with everyone really quickly."
Emma Willis is back to host the competition with solo singers, duos and trios once again singing in the hope of turning chairs.
The Voice UK 2020 starts Saturday, January 4th at 8:30PM on ITV.