It comes after Robbie Williams and Ayda Williams announced they had quit The X Factor.
The husband and wife pair have revealed this week they were stepping down from the series after one year.
They blamed their busy schedules in a post on social media, writing on Instagram: "We're gutted to tell you we won't be able to return to The X Factor this year.
"We will be working on projects with Simon going forward - and as huge fans of the show, we will still be tuning into The X Factor, voting and winding Simon up!"
However sources have alleged the couple walked away from the show after failed negotiations over their fee.
They were paid a reported £10 million for last year's series and The Sun newspaper claim were hoping for a raise.
But an insider said: “Robbie and Ayda were already on a £10million deal and there was no way we were going to be able to increase that sort of money.
“They haven’t fallen out with Simon – they were all on holiday together recently – but there are business realities at play."
Alongside Robbie and Ayda leaving, Louis Tomlinson has admitted he may not be back on the show either.
He said recently: "I had an amazing experience last year but if it's going to get in the way of my mission for this year, which is releasing the album, it's probably going to be less likely."
In the wake of the latest news, the odds of Louis Walsh making a sensational return to the X Factor judging panel have been slashed by bookmakers Ladbrokes.
Former Saturdays star and ITV regular Rochelle Humes finds herself chalked up at 4/1 for a spot on the panel, and should producers fancy another married couple working together 6/1 says Marvin returns to his X Factor roots, only behind the desk this time.
Jessica Bridge of Ladbrokes said: "Punters would love to see Louis Walsh return to the show, but it wouldn't be a huge surprise to see ITV bosses replace Robbie and Ayda with another successful husband and wife duo in Rochelle and Marvin."
The X Factor airs on ITV.
The series typically launches in September.