And she isn't exactly his biggest fan, either.
There are rumours that Cheryl could be back on the panel next year, in place of Nicole Scherzinger who has hinted she will quit.
Cheryl's partner Liam Payne spilled at the weekend: "Cheryl and Simon spoke about her going back next year. Obviously Simon is desperate to get her back."
Ahead of Cheryl's return to the show as Simon's guest at judges' houses, Liam added: "Cheryl and Simon have such a lovely relationship. Obviously she's been off for quite a long time with [their son] Bear but it was great for her to get back out and do the Judges' Houses."
Cheryl first joined the show in 2008 before leaving in 2011 for the American version. The new mum then returned in 2014 before departing again last year.
Speaking from judges' houses, Cheryl and Simon both suggested a third stint could be on the cards - but only after enjoying an extended maternity leave.
Cheryl explained: “I want to be with Bear a little bit longer. It has just been me and him and I’m just loving being a mother at the moment. He is only six months old.
"I’ve got time - I’ve got that luxury lifestyle and some mothers don’t have that luxury and have to get back to work. I’m just taking it as I can but I do love the show.”
Louis has confirmed rumours that Nicole may leave - but Cheryl isn't his number 1 choice to replace her.
I’m signed for next year, so is Sharon. Let’s see about Nicole," he said. “I would bring in Mel B, I love her, she’s a survivor. I love Nicole a little bit more, she’s my favourite judge.”
Meanwhile, Cheryl remarked on The X Factor's current line up: “You’ve got some people that have experienced everything in there from being on stage to being the manager.
"And then you’ve got Louis."
For now, The X Factor continues Saturday and Sunday nights on ITV.