X Factor boss Simon Cowell reportedly wants Gary Barlow fired from the show, blaming him for the ratings dip.
Simon apparently already has another man lined up to replace Gary, and didn't want him on the panel this year.
According to The Times, ITV bosses had overuled Simon to get Gary back on the show for a second series, but with ratings falling, it seems that the channel can no longer defend the choice to keep him on board.
"Cowell doesn't think that he knows enough about who to choose from the auditions. The one thing he did after the 2010 show was to create One Direction, who are now the biggest pop group in the world," an insider told the newspaper.
The X Factor 2012 final is set to be one of the least watched ever, with last weekend's semi-final down over two million viewers on last year.
The source continued: "The original X Factor was zeitgeisty - we were in an age where everyone was being too PC. When he told people the truth about how awful they were, people warmed to him. People also believed he really had the ability to turn someone into a big star."
Reflecting back on the series, Gary told the official ITV1 X Factor website this week: "The X Factor is much more fun this year. There has been shedloads of drama! Last year was a learning curve for me.
"This year, I’ve been less nervous and I’m taking things in my stride. I toured during last year’s show. In hindsight, I took on too much and was knackered the whole time."
Speaking to The Times, the insider added: "Gary still hasn't decided what he's going to do next year, but he doesn't need to do this. He has a day job. Whatever happens, he won't be at home twiddling his thumbs.'
However a show source told the Daily Mail: "Nobody's thinking about next year yet. Everyone's just concentrating on this weekend - and making sure it's a great finale.
"To suggest anyone else has been lined up at the moment just isn't the case - and wildly premature."
We're not sure how Gary alone can be blamed for this year's flop in the ratings.
The biggest problem has been the overall lack in talent in all of the finalists, together with some bizarre production choices.
Not letting singers like James Arthur and Ella Henderson to perform their own tracks is one mistake that's certainly not helped the series.
The X Factor 2012 continues tonight with the live final results in Manchester from 7:40PM on ITV1.
Jahmene Douglas and James Arthur will be battling it out to be crowned winner, while there will be performances from One Direction, Rhianna and Emeli Sande.