Robbie says Simon asked him to be part of the show way back in 2004, long before finally signing up for this year's series.
The former Take That member joins wife Ayda Williams on the panel, and admitted he may not have said yes without her.
But Robbie added in an interview with Daily Mail's You magazine: "Id always been busy – album, promo tour, album.
"But I’m 44 now and I’m looking for some new ventures. I’m thinking, 'Oh, who knew, I’m a TV personality now.' I’m seeing the next 20 years open up in front of me."
He confessed: "It’s a big risk, putting yourself on mainstream television every week. For somebody who can put his foot in it – unintentionally – it’s a very scary prospect.
"But fun at the same time. It’s like walking the tightrope, though there’s the fear I’m going to do something that will cause an international incident."
"It’s really important people either really love it and us – or they obsessively hate us. And I want the obsessive haters to watch too," Robbie concluded.
Simon says he's happy with the result - even if it cost a reported £10 million to sign up the husband and wife team.
"I was a little nervous before filming started. You never know what the chemistry will be like, how people will react in the chair, but these two took to it like ducks to water," he said. "We’ve got our money’s worth. Performers look up to Robbie. I think the contestants have upped their game this year, performing for him."
"[Ayda's] smart. She’s funny. She’s got a great attitude. I think audiences will love her."
Alongside Robbie, Ayda and Simon, One Direction's Louis Tomlinson is also new to the panel this year, while Sharon Osborne will rejoin the show for the live shows.
The X Factor 2018 launches Saturday, September 1 on ITV.