Friends of the singer, who joined the show's new line up last summer, have claimed that Tulisa is worrying about her future on the show.
Simon has made no secret that he'd love to see Cheryl back on the reality series, but Cheryl herself has remained coy about the subject.
Pals of Tulisa’s however suggest that she's already made up her own mind.
"Tulisa's told friends that if Cheryl thinks she's getting her job back, she'll have a war on her hands," one insider tells Closer magazine this week.
The source continued: "She says she knows there would only ever be room for one of them on the judging panel, as they're seen as so similar.
"Cheryl's considered part of The X Factor family and a lot of the people consider it her role."
But the insider was keen to point out that no official talks between either Tulisa or Cheryl had taken place, adding: "But formal discussions haven’t even taken place yet, so who knows what could happen."
Earlier this week however Gary Barlow confirmed he was in talks with ITV bosses to return to the show, but refused to comment on whether he'd signed any contracts yet.
He said only: "I started talks two weeks before the last one finished. We are talking.”
Open auditions for The X Factor 2012 kick off next month, with the show returning to screens in the summer.