Earlier this week, the Daily Star reported that the tycoon was 'desperate' for the 'We Found Love' singer to sign up, in the hope that her controversial lifestyle would bring it extra publicity.
However, Simon rubbished the rumours while chatting to the newspaper on the red carpet at last night's gala premiere of I Can't Sing!, a new musical based on the show.
"A lot of the stuff I've read about in the press... I haven't spoken to these people," he commented.
Other big-name stars that have been linked to The X Factor in recent weeks include Miley Cyrus, Demi Lovato, Rita Ora and Ellie Goulding.
Meanwhile, the 54-year-old gave his backing to Louis, who's the only person to have appeared on all ten editions of the talent contest.
"I would love him to come back and do the show, it wouldn't be the same without him," Simon said of the Irish music mogul.
So far, only Simon and Cheryl Cole have been confirmed for this year's series, with both making big comebacks after three-year absences.
Louis is widely expected to return, but has yet to sign a new contract.
Open auditions for The X Factor 2014 will take place around the country next month.