The X Factor boss reckons that singers don't make good judges on the show, claiming that they're 'too selfish'.
Cheryl, Rita and Nick Grimshaw were all dropped from the panel this year and replaced by Louis Walsh, Sharon Osbourne and Nicole Scherzinger.
Chatting to the Daily Star on Sunday newspaper, Simon said: “When you’re sitting as a judge and you’re judging these shows you have to like people. If you don’t, you can’t do this job. It’s as simple as that."
He explained: “I mean, this is one of the issues I have at the moment when you put too many singers on a judging panel.
“Singers don’t want to find other stars. It’s just not in their DNA. They only think about themselves. That is the nature of being a solo artist. I am a cold hearted A&R guy, record. I own a record label. So the only thing I’m interested in is finding a star.
“Now, you put a singer in my seat and somebody ten years younger than her comes on and is better than her, she does not want her to do well. It’s a fact.”
Simon went on to rave about the return of Louis Walsh on the new panel, claiming The X Factor just wouldn't be the same without him.
"I could not have done X Factor without Louis," Simon said, "He has been part of the show for 10 years.
“We were friends when we first started and mates before that. I like working with Louis. He makes me laugh.”
The X Factor 2016 airs Saturday and Sunday nights on ITV.