There was drama moments before last night's live show when it was revealed the duo would not be performing.
Host Dermot O'Leary said at the top of the episode: "Sadly, due to unforeseen circumstances, Brooks Way will not be performing tonight.”
Now it's been reported the pair will NOT be returning.
Why did Brooks Way leave The X Factor?
The Daily Mirror claims that the decision was made earlier this evening following crisis talks with producers.
"In the past 24 hours X Factor have had Brooks Way on lockdown whilst they try and digest everything that has been written about them in the press," a source said. "The show was desperately trying to find a way for the brothers to stay, they are very aware of the appeal they had to the audience."
The insider added: "It's obvious that the boys wanted to stay on the show, however, their family did also want to pull them from the competition. Ultimately it was the show's decision to axe them."
The Sun on Sunday newspaper this morning reported on allegations concerning Josh and his ex-girlfriend.
A spokesperson for The X Factor has yet to confirm the news but said previously: "Due to the serious nature of the information which has come to light concerning one of Brooks Way, the group will not be performing in Saturday night’s show pending further inquiries.”
The decision to suspend Brooks Way - previously known as The Brooks - came with just hours to go before last night's show.
The boys were seen getting their make up done backstage at 5PM as they prepared to sing Cake By The Ocean.
"Not only are the brothers heartbroken and in tears but all the contestants are totally devastated," a insider revealed.
The X Factor 2016 continues tonight at 8PM on ITV with the first results show.