The music mogul had wanted the current Voice UK coach on the series last year but she instead chose to join The Voice UK.
Rita said earlier this year it was her fellow coaches - and Will.i.am in particular - who made her pick the BBC series.
“What makes this show so good is that we [the judges] each see the contestants from such different perspectives," Rita said ahead of the fourth series. “I knew that what I thought would never be similar to what the guys think and vice versa."
However Rita's debut series didn't end that well as she was left with no acts in the final and this month refused to confirm whether or not she wanted to return.
X Factor boss Simon reckons Rita's "fair game" to try and persuade (again) to join The X Factor.
“I saw Rita on The Voice and I thought she was very good," Simon explained to the Daily Star newspaper. "When it comes to asking her over, you know, it’s fair game."
He revealed: “We were calling Alesha for two years to come and join one of our shows.
"I think they have to want to come over in the first place.”
It's been said that Simon 'can't decide' the X Factor judges, keeping the would-be panelists in the dark just as the rest of us.
"Many who are involved in the show feel that [Simon Cowell] is playing mind games," a source claimed. "No-one is safe from the Cowell axe."
The X Factor will air on ITV later this year.