The Apprentice boss took to Twitter to complain about the N-Dubz star being a judge on the show.
He moaned on the social networking website: “Can you tell me who the hell this Tulisa bird is on X factor,she’s in the celeb section of the sun and mirror every day. What has she done?”
He added: “This Tulisa talks nonsense on X factor,what qualifications does she have to judge singing talent.Please advise where I am missing the point”
Tulisa didn't take long to hit back however, replying to Lord Sugar: “stick to judging on the Apprentice & I’ll stick to what I know MUSIC #itsaNOfromme”
Olly this week defended Tulisa over Lord Sugar's remarks, saying she had "every right" to be a judge on the show.
"I don't know why he said it," Olly said.
Speaking at the V Festival, where both he and Tulisa were performing, Olly added: "Tulisa has every right to be on the judging panel. She has been in the music industry for years and has had loads of success.
"She is doing what she knows about. When it comes to the work place stuff, that's Sir Alan's area, so she leaves that alone."
Olly will return to The Xtra Factor alongside Caroline Flack on ITV2 when the live shows start in October.