We'll see this weekend former X Factor judge Cheryl back on the show, helping out Gary at his judges' houses.
Cheryl joined the Take That star at a stately home in England to decide on which of the three overs to take through to the live shows.
But Simon wasn't happy with the choice, angry that Gary went over his head!
"Simon and Gary have never got on," a source claims. "It's no secret that Simon didn't even want him to the panel to begin with.
"So when got Cheryl ins without even telling Simon, did did not go down well at all. Simon said it was undermining him a the show's key decision maker."
Speaking to Star magazine, the source claimed that Simon wanted to bring Cheryl back himself!
The supposed insider explained: "It really got Simon's back up that Gary got Cheryl back on the show, he wanted to bring her back himself.
"He felt that it was just to stick two fingers up at him and he was angry about how secretive it was."
Cheryl is also due to perform on the live shows, which kick off on October 6 on ITV1.
The X Factor 2012 continues this Saturday and Sunday on ITV1.
Browse pictures of Simon and Gary on The X Factor below...