The singer has reportedly been approached to join the panel this year after appearing as a guest judge on last year’s series.
But despite apparently being on Simon's "wishlist" of judges for the show, the singer thinks her career is too new to take on the series.
"I actually met Simon finally at the Brits after all the things we did for X Factor," she said this week. "He came up to me, he said, 'Thank you so much for doing that for the show'. I said 'Yeah, of course, it's fine', and then he looked at me, he gave me that look like there's something coming but he never asked me."
Speaking to DigitalSpy, Rita added: "I think right now I don't think it would be the right time to do anything X Factor.
"I'd love to come back in and out and be a guest judge but if I was to take on the full role it would come from a place where I feel I'm satisfied with my career and right now I've got a lot to do."
The X Factor 2013 returns in the summer and it’s widely speculated that both Gary Barlow and Tulisa will be gone.
As well as Rita, Sharon Osbourne, Robbie Williams and Cheryl Cole have all been touted as possible replacements.