In last Saturday evening's second live show, Brendan and celeb partner Charlotte Hawkins were dancing the Cha Cha to Sugar by Maroon 5.
Charlotte made a mistake early on and it led to "a few stumbles", to quote Brendan.
The judges didn't hold back with their criticisms and the pair ended with just 12 marks, but managed to avoid the dance off thanks to the viewer vote.
Speaking on It Takes Two in the week, Brendan has his say as Zoe Ball quizzed him on the judges' comments.
Last weekend, Shirley had suggested Brendan could have made the routine easier but he hit back: "I think she could keep those opinions to herself.
"That was a very simple routine, it was a classic, cha cha. It was basic choreography, I put in two difficult steps..."
Brendan, a judge on the New Zealand version of the show Dancing with the Stars, remarked: "If Shirley wants to swap places with me I'm quite happy to judge her, I know how to do that."
He went on to tell host Zoe: "It disappoints me because as the pros you almost can't do the right thing. If you give too much, it's too much, if you give too little, it's not enough. She did say in the opening show, 'I want to be excited, I've seen everything, show me something I haven't seen before', now it's 'you made it too hard'.
"It's frustrating, but I don't worry about them, I'm focused on us."
Charlotte added: "You want to push yourself and you don't want to go out there and do something ordinary."
Strictly Come Dancing 2017 continues this evening and Sunday night on BBC One with its Movies themed episode.