The 71-year-old admitted he feared making the wrong choices on the show, including when it comes to the battle rounds next month.
The next stage of the competition, following the blind auditions, sees each of the coaches send five of the singers in their team home.
"You are throwing yourself out there and making decisions. People at home could be thinking, 'What the f***? Why would he do that? What's the matter — is he deaf?'," Tom explained this week.
He went on: "When you are out and about on the Tube, or in a cab, the drivers are going to go, 'Hey Tom, the show is s***'. We have to make the show as good as we possibly can and whatever is thrown against you, fight back."
The show launched to over 8 million viewers on Saturday night, but still lost out to Britain's Got Talent in the Saturday night ratings battle.
Sir Tom said: "The ratings war is out of our hands.
"As long as we get a fair crack of the whip, then it is up to the viewers. At the end of the day it is always up to the viewers."
The Voice UK continues on Saturday night on BBC One.