Matt made his name when he fronted ITV2 spin-off The Xtra Factor with Caroline Flack in 2013, but he left after one series.
But it seems he's happy that he's no longer involved, saying the Simon Cowell talent contest is "quite horrible" - and that Xtra has become a "poisoned chalice" by changing hosts every year.
"[The X Factor]’s a sinking ship. Whatever happens on that show you’re going to get absolutely mauled for it," Matt told the Radio Times recently.
"People are bored of it. They don’t want that negative ‘go sing for your life/beg to stay’ and all that. It’s quite horrible when you watch it back.
"I think now Xtra’s a bit of a poisoned chalice. I left and every year they’ve replaced them now. It did loads of good stuff for me, but I’ve got no desire to go back."
It's no surprise then that Matt prefers The Voice UK, which by contrast has a reputation for offering its contestants constructive criticism.
"It’s much more in the same vein of Strictly in that it’s ‘look how good these people are!’ rather than begging and crying and all that kind of stuff," the comedian commented. "I really like that about it. I think Emma Willis is unbelievable on it. It’s just a much nicer feel."
And with The Voice moving to ITV next year, Matt says he'd be up for joining its new companion show.
"I’ve done my fair share of spin-off shows, they’re really good fun. The Voice one is what I’d love," Matt said. "I’ve told my agent, obviously. The last spin-off I did on ITV I got fired from, so let’s see if they give me another chance!"
ITV has confirmed that it will air three series of The Voice from 2017, accompanied by a brand new spin-off on ITV2.
Matt - who's now part of the line-up of the relaunched Virgin Radio - has also claimed that Caroline "took a bullet" for Olly Murs when they presented The X Factor last year.