Piers was previously a judge on the show from its first series up until 2010 as well as appearing on the American version.
In a new interview this week, he says BGT producers asked him to sit in for Simon on the most recent series due to the music mogul's back injury.
But Piers explained he'd only return to the show if David Walliams was given the axe.
"I do have conversations with BGT every now and again, particularly after Simon had his injury," he told the Sunday Mirror. “But I couldn’t work with Walliams. I think he is a duplicitous, treacherous, piece of work.
"He is one of those people who says one thing to your face and then sticks it to you behind your back.
"If he was sitting there I wouldn’t be interested. But if he wasn’t sitting there, who knows?”
Piers added: “I will come back to BGT if the money is right and if they guarantee Walliams gets fired. I might even make it a contractual clause that I get to be the one who tells him he is fired.
"If I was him I would stick to writing his children’s books.”
The comments follow claims that producers are keen to entice Piers back to the panel in place of David who joined the show in 2012.
The Sun newspaper reported that current contracts mean that any potential switch won't take place until 2022.
"Britain’s Got Talent is one of the few Saturday night shows which continues to shine and stay relevant," a source told the tabloid. “But all shows need refreshing, and Piers will certainly do that. He has been approached by two senior execs and, for his part, has said he would do it if the terms are right.
“He will give the show back its edge and provide those brilliant water cooler moments. He will revitalise the series.”
However a spokesperson for the show insisted: "Unsurprisingly, during these times, it won’t come as a shock to the public to hear that no such discussions have taken place regarding BGT 2022."
For now, the Britain's Got Talent panel is made up of David, Alesha Dixon and Amanda Holden, who has appeared on the show since the first series.
Britain's Got Talent will be back for what will be its fifteenth series in 2021.