The Daily Star claims that the 'We Found Love' singer is top of a list of 'sexy girls' being targeted for the new series.
Simon is thought to favour Rihanna as her often controversial antics could help the show steal the spotlight from BBC rival Strictly Come Dancing, which has thrashed it in the ratings in recent years.
His partner in crime Cheryl Cole - who apparently gets to help choose the remaining judges as part of her comeback deal - is also said to be 'desperate' for Rihanna to join.
An insider told the newspaper: "Simon and Cheryl have two months to find someone to sit in that chair and they want to get the right person.
"This year is an important one for The X Factor. Simon can’t be complacent and has to come back with a real A Team. Rihanna would be a shock selection because she is not an obvious choice for a family show – but that’s what he wants."
Rihanna was seen being bombarded by fans as she arrived at Heathrow Airport yesterday, sparking rumours that she's about to attend a meeting with Simon.
The Barbadian singer has sold over 150 million records worldwide and has racked up eight no.1 singles and four no.1 albums in the UK. However, she's become just as famous for her lifestyle in recent years, attracting criticism for frequently smoking marijuana.
So far, Simon and Cheryl are the only two judges confirmed for this year's X Factor, with both returning after a three year absence.
Show stalwart Louis Walsh - who has taken part in every series to date - is expected to be back, but is yet to officially announce his involvement.
The X Factor 2014 hits our screens in the autumn, with open auditions taking place next month.