It means only Simon Cowell is set to remain on the panel from last year's show in what's looking to be another huge shake up to the series.
Nick Grimshaw was the first to step down from the judges' desk at the start of 2016, shortly followed by Cheryl back in April.
Taking to Twitter this morning, Rita confirmed reports that she had left the show as well.
She tweeted: "I had a ball on @TheXFactor last year and will miss the team."
And in a message to Simon Cowell, Rita added: "Can't wait to work with you again... I'll be round for dinner soon. Thank you for the experience & love X"
A spokesperson for The X Factor added: "Rita brought a great energy to the show last year and did a brilliant job mentoring the girls' category, leading Louisa to victory.
“We wish her all the best with her music and film plans for the rest of the year.”
It was reported last night that Rita had quit due to clashing schedules.
A source told The Sun newspaper: “She was very keen to do the show but at the end of the day it didn’t work out with her other commitments around the world in music and acting.
“It means we’re going to have to go through the biggest shake-up in the history of X Factor, but maybe it will turn out for the best.”
The tabloid claimed that Rita's plans for a new album as well as work on the new 50 Shades of Grey movie made appearing on The X Factor "not logistically possible".
Alongside the judges, hosts Caroline Flack and Olly Murs have also gone, with Dermot O'Leary already confirmed to replace them.
As for who will take over from Rita, Cheryl and Nick we could see a throwback to 2013 with Louis Walsh, Nicole Scherzinger and Sharon Osbourne all returning, with Simon taking Gary Barlow's place.
But as yet nothing has been confirmed!
Stick with us for all the X Factor latest!
The X Factor 2016 will air on ITV in the autumn.