The music mogul's cheeky line of questioning left one female act unhappy - and showing him just how much.
A source at the try outs told the Daily Mirror newspaper: “The woman looked like she was in her 60s but wasn’t willing to share her age with the judges.
“Simon kept pushing her for a number and so she ran up to him on the panel and slammed her hands down on his arms, giving him a bit of a slap."
They added: “She told him he should never ask a lady her age, but the other judges were pretty shocked at her slapping Simon.”
Meanwhile, another act to appear before the panel had a rather familiar name.
Harrison Styles is no doubt hoping to have as much as success as a certain One Direction star.
The source said: “He was telling the judges he wanted to prove he was a star like his namesake. There was a lot of amusement about his name but Harrison was very serious about his singing.”
Auditions for the new series were held n Manchester at the weekend with guest judge Alesha Dixon.
The current Britain's Got Talent judge stepped in for Nicole Scherzinger at the latest try outs due to a diary clash.
A spokesperson for The X Factor confirmed: “Alesha is guest-judging for the first day of the Manchester auditions as Nicole is unable to film that day due to a previous diary commitment.”
However Nicole also missed the first half of filming on her planned return, reports say.
A source told The Sun newspaper: “Her flight was delayed and the airline lost her cases.”
The X Factor 2017 is expected to launch at the end of August on ITV.