While Gary and Rylan’s banter is usually quite playful, Gary seemed pretty angry with the Essex performer this week, following the exit of Ella Henderson on Sunday night.
Gary said: “I’m serious. What are we going to do about Rylan?
Speaking at a gig in aid for Mencap on Tuesday, Gary continued: “I might have to be honest on Saturday night and go: ‘Rylan, I thought that was absolute s****’.”
However Rylan hit back at Gary on Twitter by bitching about Christopher Maloney, Gary's last remaining act.
"I'd rather be 'absolute s***' than just stand there and sing the same old stuff every week on the spot....... #boreme @GBarlowOfficial," he tweeted.
"And I entered the x factor.... NOT the voice..... Cheers......"
On a rant, Rylan then went to hit out at his other haters on the social networking website.
He wrote: "Don't think people understand some of the stuff that I get sent day in and day out. Some of you should be locked away for your comments"
The Essex performer later added: "I'm a 24 year old normal boy who has auditioned for Xfactor, never come from money, my mother is seriously ill, and not one have I used that.
"As a sob story or a reason to validate my place on the show, I would rather be judged on the reason why I'm actually on the show to sing."
The X Factor 2012 continues this Saturday on ITV1 and ITV1HD, with the final five acts sing two tracks each with Abba and Motown themes.
Sunday's results show will see one more act get the axe, and there will be performances from Rhianna, Tulisa and Bruno Mars.