The former X Factor finalist has been tipped to replace Dermot O'Leary on this year's show after it was confirmed last week we'd be getting a new host.
Dermot quit the show after eight years, saying it was time to 'move on'.
"It's so sad to see him leave," Olly said today.
The singer, who auditioned in 2009 and made it to the final, said: "Dermot was there for me from day one. Dermot is a personal friend of mine and is someone I get on really well with.
"I have known him since 2009 when I did the show as a contestant. It is obviously a massive shame, he has done an incredible eight years but I think it is now time for a new chapter."
There have been rumours that Olly is in talks to replace Dermot, with some reports suggesting that his former Xtra Factor co-host Caroline Flack could be joining him.
Olly said today: "It's rumours. Everyone is speculating about it. I have presented the Xtra Factor before with Caroline and I loved it."
Speaking at the launch of his new waxwork at Blackpool's Madame Tussauds, Olly added: "Caroline is great. We worked together years ago and we loved it. Every year there is speculation that I am going to do the X Factor, as a judge or presenter, and I just ignore it."