The twin brothers - Josh and Kyle - lost their place on the show over the weekend.
A spokesperson for the show confirmed today: "Following the allegations relating to Josh Brooks, we have decided by mutual consent that Brooks Way should leave the competition."
In a statement, Josh added: "Due to events in my personal life I've decided that it's best not to be on the show at this time.
"While not everything that has been claimed is true, I would like to apologise to everyone involved."
The pair, who yesterday hit out at "false allegations" made against them, looked all smiles as they tweeted the above selfie this evening.
The boys went on to congratulate Four of Diamonds who will replace them in the competiton.
"So so happy for you girls! You deserve it," the Brooks wrote on Twitter.
"Thank you so much boys," responded the girls.
ITV confirmed this evening that Four of Diamonds would be joining this year's X Factor live show line up.
Mentor Louis Walsh said: "I was really impressed with the girls at Judges’ Houses but only had three spots to fill for the Live Shows.
"At the time a lot of viewers were disappointed that I didn’t put the girls through, now that a place has opened up in my category, I’ve listened to the fans and am delighted to be bringing the girls back. They're a great girl group and I'm hoping will take me all the way to the final."
Speaking about their return to the show, Four Of Diamonds said, "We are so excited to be back on The X Factor we’re really looking forward to performing live, working hard and showing everyone what we’re made of."
The X Factor 2016 live shows continue this Saturday and Sunday night on ITV.