Kyle and Josh Brooks were suspended from the contest hours before Saturday's live show after serious allegations regarding Josh emerged.
A spokesperson for the show said on its website: "Following the allegations relating to Josh Brooks, we have decided by mutual consent that Brooks Way should leave the competition."
In a statement, Josh said: "Due to events in my personal life I've decided that it's best not to be on the show at this time.
"While not everything that has been claimed is true, I would like to apologise to everyone involved."
One of Louis Walsh's groups will be brought back as a replacement, with further details to come "in due course".
Why did Brooks Way leave The X Factor?
The Sun on Sunday newspaper yesterday reported a number of serious allegations concerning Josh and his ex-girlfriend.
However, Josh insisted that some of the accusations are "false" in a message posted on Brooks Way's Twitter page yesterday.
"Today, we have had a tough day with a false allegation," the tweet said. "Don't believe everything you read, both Amelia and Josh will be stating the truth."
X Factor producers cut Brooks Way's performance from the first live show after becoming aware of the situation.
They were featured in an introduction VT on Saturday - but were not referred to whatsoever in last night's results.
Which group should replace The Brooks? You can vote in our poll below...