The current series is only the second in the ITV talent contest's fifteen-year history not to include Louis.
Instead, bosses refreshed the judging panel with a new Louis - One Direction star Louis Tomlinson - as well as pop superstar Robbie Williams and his wife Ayda.
And while Cowell has been full of praise for his new colleagues, the absence of the Irish music manager has not gone unnoticed.
At a recent Health Lottery charity event, Lizzie Cundy asked the Syco boss if his was missing Louis.
"Yes I am," he confessed, before suggesting a way he could return to The X Factor.
"But you know, he still does Ireland's Got Talent and he can come back... as a contestant."
Cowell even cheekily suggested he could create a new category on the show especially for his former right-hand man.
"We have the Overs category so we would have to have the 'Older and Older' category. For the over 80s," he joked.
Louis wouldn't quite fit in to that bracket yet, as he's only 66 years young.
In addition to his Got Talent role, Louis is currently busy overseeing the reunion of much-loved boyband Westlife.
The lads have announced a new arena tour to mark their 20th anniversary, and are also expected to release new music.
While he's no longer on the show, Louis is still watching The X Factor and recently praised Robbie.
He also took a swipe at rival show Strictly Come Dancing, calling it television for "older people".
The X Factor continues on Saturday at 8.35pm on ITV, when the contestants will face Fright Night.