Louis feels like part of the furniture on the ITV show - but Simon has tried to ditch him three times.
His first go in 2007 backfired, and the Irishman was reinstated within a matter of weeks.
Further attempts in 2015 and 2018 were slightly more successful, but both times Louis returned after just a series away.
Now sat at the judges' desk once again, Louis believes he's essential to the talent contest - as evidenced by Simon's son, Eric.
"He may not want to admit it but deep down he knew he needed me back," he said in an interview with The Sun. "He always realises in the end.
"Even his son wanted me back. Simon phoned me and said Eric told him, ‘Can you bring back Uncle Louis?’"
Never one to hold his thoughts back, the Westlife mastermind couldn't resist having a dig at the celebs who replaced him.
While he thought Robbie Williams did a "great" job last year, he isn't sure Simon was right to hire the 'Rock DJ' singer's wife, Ayda Williams.
"Most people just didn’t know her, and the ITV audience have to know you," he commented.
"She was thrown in at the deep end and she’s a very funny woman, and they’re a great couple together, they really are. But I just don’t think the public warmed to it."
However, Louis was even less fond of Nick Grimshaw, who sat in his chair in 2015.
"Nick Grimshaw? Oh my God, let’s not go there, I’ve forgotten about it," he remarked. "That was not a good booking. No, no."
In addition to the Celebrity edition, Louis is due to remain for a planned 'All Stars' special later this year.
Nicole Scherzinger won't be sticking around, with rumours suggesting Paula Abdul might take over her spot.
The first X Factor Celebrity live show airs tonight at 8.20pm on ITV, with Walsh mentoring the Unders.