The presence of the "genuine urban artist" in the live shows has rubbed some up the wrong way - especially after some of the favourites to win failed to make it.
And in comments likely to further rile disgruntled viewers, Simon has admitted he thinks Honey G's music would flop - at least at the moment.
When one Xtra Factor viewer asked if she can sell records, the mogul responded: "The answer is no.
"But that's why we do the show, because we kind of embrace this I suppose. That's how you build a following.
"It's a little early, this is her first week, but I've gotta tell you, she won me over tonight."
Fortunately for Honey, she has plenty fans too, who were thrilled by her version of 2Pac's 'California Love' last night.
Simon, the only judge to say 'no' at her room audition, reacted: "Here's the weird thing, I loved it. I just really like you. I thought you weren't serious when you came in but now I realise you are."
We reckon she did North Weezy proud!
Tune in to ITV tonight at 8.00pm for the first results show of the series.