But fans of Nicole needn't worry as Simon Cowell hasn't had a sudden change of mind about this year's panel.
Instead, Mel B has been drafted in as a guest judge to stand in for Nicole at the upcoming try outs in London.
Simon confirmed at the start of the month that he would be joined by Nicole, Sharon Osbourne and Louis Walsh on this year's panel of 'fan favourites'.
While Nicole snubbed the chance to appear in a Broadway production of Cats to join the show she couldn't clear her schedule completely.
Therefore America's Got Talent judge Mel will fly over to take a seat for a third appearance on The X Factor.
Mel was previously a guest judge in 2012 and also appeared as a full time judge for the 2014 series.
“This is a diary clash that Nicole had booked in before she agreed to do X Factor and couldn’t change," a source revealed this week.
They told The Sun newspaper: “Mel has been working with Simon as a judge on America’s Got Talent so was available to cover.
“He also knows she’ll come in and take no prisoners so is the perfect temporary replacement for the day.”
The X Factor will air on ITV later this year.