Dancers Libby and Charlie will replace them in Friday's semi-final, it's been revealed.
A group of three masked magicians, Brotherhood stunned the judges at their audition with a terrifying sword trick.
Host Dec Donnelly secretly placed an upright blade under one of four cardboard boxes - before they miraculously jumped on the three that were empty.
The heartstopping stunt won them a place in the top 40 acts, and they were due to compete in tonight's fifth semi-final.
However, in a shock move, Brotherhood have today withdrawn from the line-up.
Instead, young dance duo Libby and Charlie will take their place, having originally failed to make the cut.
A BGT spokesperson confirmed the news, saying: "Brotherhood have had to withdraw from the Britain's Got Talent Live Semi-Finals.
"Libby and Charlie will now be performing on Friday nights show."
It's said Brotherhood stepped down from the competition for health and safety reasons.
A show source told The Mirror: "The health and safety of all the performers is a huge priority to the show.
"Producers will only allow acts to take part who they are certain are adequately prepared"
Brotherhood said the decision was due to circumstances "out of our hands".
In a series of clips uploaded to their Instagram page, the three piece showed how their stunt involved one member being trapped in a cage set on fire.
A second part sees another member underwater, locked in a padlocked container.
In an Instagram post, they wrote: "We’ve worked for the last 4 months on an act so ambitious and out there that we’ve put our sanity, blood, sweat, fire and tears on the line.
"Through circumstance and bad luck out of our hands sadly we aren’t in a position where we can deliver.
"It kills me that we’ve failed and that we will no longer be in 2019’s @bgt competition. We are truly gutted that we won’t get to be in the fight but we aimed for the stars and fell short.
"It’s been an absolutely incredible ride for BGT this year and I wish all the acts still In the competition the best of luck and success and that a magic act can bring home the crown again."
The Britain's Got Talent semi-finals air tonight and tomorrow at 7.30pm and 9.30pm on ITV.