Ayda has joined the ITV talent contest's judging panel this year alongside her pop superstar husband Robbie Williams.
Her involvement has raised a few eyebrows due to her lack of obvious music industry experience.
However, the American actress has been well received by audiences during filming - and she has a huge fan in Simon.
In fact, the Syco boss is so impressed that he couldn't resist having a cheeky dig at former X Factor stalwart Louis in an interview on ITV's Lorraine.
Simon said: "She’s been annoyingly, really, really, really good, to the point where I felt like Louis Walsh halfway through the show. I’m the invisible one!"
The fourth judge on the new series, One Direction star Louis Tomlinson, echoed his comments.
"Amazing actually, let me second that," he grinned.
Explaining why he decided to shake up the show this year, Simon continued: "Sometimes you’ve just gotta make a change, and a radical change.
"I always say the best panel is people you would like to have dinner with, and that’s where we’ve ended up."
The media mogul also told showbiz journalist Dan Wootton that the fun atmosphere between the judges reminded him of X Factor's very first series in 2004.
"It’s a new show, that’s the only way I can describe it," Simon commented.
"It has been the most fun I’ve had in years, it really does feel like when we first started off."
The X Factor returns to ITV tomorrow and Sunday at 8.00pm on ITV, with a few format changes to keep contestants on their toes.
Auditions took place exclusively in arenas this year, while the Six Chair Challenge has been spiced up with a 'Golden X' twist.