Speaking at the launch of the new BBC One series, Sir Tom hit out at the music mogul, saying he had no right to judge talent.
"Simon Cowell is a record man, not a singer," the Welsh legend declared. "I have problems with people who have never got up on a stage to sing, judging those who do.
"It’s like critics who criticise actors, never having been up on a stage in their lives. Who are they to say?"
Sir Tom revealed how Simon had offered him a job on The X Factor and Britain's Got Talent "several times", but he had always declined.
"I’ve never wanted to do one until I saw The Voice in America. I immediately thought, 'this is different'. It’s a different feel," the 71-year-old added.
The difference, Sir Tom says, is in the how the show's coaches - not judges - help the contestants.
He says: "I'm not judging; I’m coaching. It’s not about me saying, 'You’re not good enough', and being horrible about it. It’s about being tough but encouraging."
The Voice UK launches on BBC One next Saturday night.