This year Simon is ripping up the rulebook for his signature talent contest in order to give it a new lease of life.
The classic Boot Camp stage has apparently been scrapped, alongside the controversial Six Chair Challenge.
Even the usual hordes of fame-hungry hopefuls have been sent packing after the next regular series was postponed.
Instead, two spin-offs will air in its place later this year - one with celebrity contestants, the other with 'all-star' returning favourites.
Now he's plotting to go a step further with a change that could leave The X Factor almost totally unrecognisable.
The show has had the same theme song since its launch in 2004, with a remix debuting in 2011.
The tune - known for playing over the 'X' logo flying through space in the old title sequence - was co-composed by Simon himself.
However, the Syco chief has told the Mirror that he wants to replace it with an entirely new track.
Not only is he now bored with the original, he claims it is "one of the worst theme songs" he's ever heard - a comment we're sure thousands of telly fans disagree with.
"We’re scrapping the theme song. I can’t listen to that again. I will never hear it again," he commented.
"It is one of the worst theme songs I’ve ever heard.
"I’m asking around in LA to see if someone can write us something new."
Simon is also planning to give the live shows a fresh look with a brand new studio set.
"We’re going to completely redesign the set,” he confirmed.
"I haven’t decided how it will look yet, but I have met with a designer."
The set was last refreshed only two years ago, when it moved from the now-defunct Fountain Studios to LH2 Studios in London.
Simon claims the sweeping changes are necessary to prevent The X Factor from stagnating further.
"The risk is doing nothing – boring and predictable," he added.
Filming of the celebrity version is reportedly underway, although ITV remains tight-lipped on scheduling plans.