The former X Factor finalist has been widely tipped to take over the show following Dermot O'Leary's exit last month after eight years.
But Olly declared this week: “There’s no way I would even discuss doing the job on my own to replace Dermot."
He said: "Dermot is one of my best mates in the industry.
“He’s my showbiz brother. I would never do the job on my own and never try to be Dermot O’Leary.”
Speaking to The Sun newspaper, Olly went on to say he'd only ever host X Factor alongside former Xtra Factor pal Caroline Flack.
And he said that IF the pair do get the gig - as has been widely rumoured - that us viewers can expect something very different.
Olly said: “It would be completely different to what Dermot did.”
There have also been reports suggesting that Olly could end up being a judge on the show, admitting that there had been talks in previous years.
However, he suggested that the pressure of joining the panel wasn't quite for him, saying: “The judging side has a lot more scrutiny and people really turn on judges quite quickly.
“Hosting the show is completely different because it’s a little bit out of my comfort zone, even though I have presented before with Caroline.”
The X Factor returns to ITV later this year.
Browse pictures of the potential X Factor 2015 presenters below...