The Welsh singer also hit out at the show's contestants, saying that there were no "freaks" on the BBC one series.
"It upsets me when, on The X Factor, the judges are cruel to people needlessly,” the outspoken reality critic tells TV Buzz this week.
And he insists it's an opinion shared by fellow coaches Jessie J, Will.i.am and Danny O'Donoghue.
He continued: “All of us on the panel think that way. We’ve all been through that sort of rejection, whereas people on other shows haven’t – they’re just hearing it from a commercial or a punter’s point of view.”
Instead, Sir Tom claims, The Voice UK coaches are much more "professional".
He explains to the magazine this week: “Danny is an absolute riot and because he’s Irish we have a strong Celtic connection.
"Will.i.am is funny and then there’s Jessie – she’s a young girl, but she is very strong and really knows what she’s talking about. She can explain things about the voice and breathing in great detail. It’s stuff I learnt when I was younger, but you forget it all.”
He adds: “Will.i.am and Jessie do have a little banter once in a while, but Danny said to me: ‘Let them argue. We can be cool.’ It is a great combination.”
Meanwhile Sir Tom admits that he would not be fazed about auditioning for a talent show if he was just starting his career now, saying: “That’s what I do every time I stand on stage. You know that people are always going to be wondering: ‘Is Tom singing as well as he used to?’
“Even though I’ve been doing this for so long, it’s a continual learning curve. I like to do things that aren’t expected of me and love wondering what will happen next. There is no master plan, things just happen.”
The Voice UK continues on Saturday nights on BBC One.