The 'Hot Right Now' singer had originally appeared on the show as a guest judge in 2012 and was approached by producers to join the panel permanently then.
Rita said at the time: “There were talks. But if I were to take on that role it would have to be at a point when I could give it my full attention.
“I have so much to accomplish first and it’s kind of a full-time job being on X Factor.
“It’s not like a nine to five. It’s constant, for months.”
According to the Daily Mirror newspaper this morning, Rita has now once again turned down an offer to join the show after meetings with X Factor boss Simon Cowell and producers.
"Rita was formally offered the X Factor job and was very flattered, but turned it down," a source told the tabloid. “She is out in LA at the moment, having meetings, making contacts and working hard in the studio.
“Rita is bringing out a new album this summer – so that’s her priority."
That supposed pal went on to add: “She wants to focus purely on her music and not get distracted by other things. She also saw how Jessie J’s album struggled after she appeared as a coach on The Voice, and thinks the two could be connected.
Speaking in 2012, Rita insisted she would never say never to joining the show in future.
“Not that I have anything against it,” she explained. “I love it but if I was to take on a full-time role it would be at a point when I’m loads of albums in."
With Rita out it leaves Olly Murs and Robbie Williams as favourites to join this year's rumoured X Factor panel alongside Cheryl Cole, Simon Cowell and Louis Walsh.