The rapper, who has made it as far as the Quarter Finals now, has split the show's other contestants as much as viewers.
While some have praised the surprisingly "normal" performer, not everyone has nice things to say.
James Hughes, who missed out on a spot in this year's finals at Nicole Scherzinger's judges' houses, said: "I'm totally against joke acts on the show.
"It's a talent show and you're meant to be there because you have a talent in singing and performing and I don't think someone who sings the same song every week deserves to win the competition."
Speaking to The Daily Star newspaper, James complained: "Honey G should not have been in the live shows to start with."
And he added: "She's very serious and she takes herself so seriously but I don't think the X Factor is for a joke act.
"It shouldn't be for someone who wants to be famous. I don't think Honey G's in that for a record deal because let's be honest nobody's ever gonna buy that record."
So given James isn't rooting for Honey G, who is he backing?
"I think everyone thinks it's going to be Matt but I think it will be a shock," said James, tipping 5 After Midnight to take the win.
In good news for James, Honey G has been made the favourite to leave this weekend in the latest live show.
This weekend it's the Quarter Finals and, for the first time this year, the remaining top five acts will be singing two songs each.
The X Factor 2016 continues live this Saturday and Sunday night at 8PM on ITV with Louis Loves week.