The Xtra Factor host, who finished as runner up on the show in 2009, said that Simon was hoping for the winners to take over the world.
Olly has has been in the US this summer on tour with One Direction, and both of the X Factor acts have experienced success in the charts in America and around the world.
Olly told MTV: "I mean, to be honest, I did The X Factor three years ago in the UK, and it's been a real building process for me in the UK; so, I never really expected to sort of like come to America this soon.
"I'm really happy that it's happened. So, yeah, it's great. That's what Simon's vision is: he wants The X Factor to breed stars that are gonna go across the world."
He added: "It's been good so far. I'm sure there's going to be more heard in America soon."
And Olly suggested that this year was better than any for the show's contestants to get success.
He said: "I think it's the perfect time for British music to come over here. The Americans, you guys have really warmed to us, which is great, and you're liking our music which is cool.
"Obviously the success of the likes of One Direction and The Wanted and obviously Cher Lloyd, a few of them, Adele, Coldplay... to come over here off the back of that hopefully they'll like what I'm doing."