But it's not all those claims of a fix that are being looked into, rather a rather obvious F-bomb that was dropped during one of the episodes.
A total of 6 viewers made official complaints about October 23's show following the language.
Finalist Gifty Louise shouted the expletive in a live episode as the elimination results were announced.
It was as Relley C left the competition in the sing off against Four Of Diamonds, while Ryan Lawrie was saved from the bottom three thanks to the 'life line' vote twist.
In the episode, Dermot O'Leary began as usual by announcing the trio of acts in the danger zone.
First to be called safe was Samantha Lavery, prompting Gifty to shout "F**king told you!", which was caught on one of the many microphones in the studio.
Host Dermot was quick to issue an on air apology over the language, which aired before the 9PM watershed.
"You may have just heard some inappropriate language, we deeply apologise," Dermot said to cameras. “As you can imagine tensions are high.”
But it wasn't enough for some viewers with 6 taking the time to complain to TV watchdog Ofcom.
They are now investigating the issue.
"Ofcom is investigating this programme, which included the most offensive language before the 9pm watershed," a spokesperson for the regulator said.
After the results show yesterday Gifty - who was eliminated the next week - laughed off the incident and tweeted: "I didn't even notice I did it 😂😂😂😂😂😂😂😂😂😂😂😂"
The X Factor is back next Saturday and Sunday night on ITV at 8PM.