This year Simon will be joined by new X Factor 2018 judges Ayda Williams, Robbie Williams and Louis Tomlinson for the 15th series.
They appeared together for the first time on Tuesday evening at a press conference in London.
Speaking at the event, Simon explained why Louis Walsh had been let go.
"Louis, we wouldn't be sitting here without him," Simon praised. "Louis has been brilliant on the show, it just got to a point where I said, 'Something's got to change.'"
He told the Daily Star newspaper: "And he does Ireland's Got Talent for us. The show does really well for us over there, he's doing it again, he'll probably do another show with us."
Talking earlier this year, Louis said of his exit: "I’ve had a fantastic 13 years on The X Factor but the show needs a change and I’m ready to leave.
“I’m looking forward to another series of Ireland’s Got Talent, going back to my day job as a manager, writing my memoirs and seeing the world!”
Meanwhile, Sharon Osbourne will be back - but only taking part in the live shows.
Simon said: "Sharon, we wanted a different role and we thought an impartial judge on the live shows would be more interesting."
Sharon Osbourne added: “I can’t wait to get back to the judges’ desk. I love the new format.”
Discussing the new judges, Simon declared: "This new panel is all about optimism, everyone has a real passion for finding a new star. The three new judges bring a whole new energy and I’m excited.”
The X Factor will air on ITV in the autumn.