Olly Murs and co-host Caroline Flack were late signings to the series, booked with just days before auditions were due to start.
Olly boasted to The Sun: "Lots of people were sceptical. And I saw a lot of negative tweets when I got the job.
"People were saying, 'He's not going to be good enough' and 'It'll be great to watch Olly sink on camera'. It was great to prove them wrong... [I've] put my own stamp on the show."
But the singer - who finished the show as runner up in 2009 - ruled out a permanent move to hosting.
"I like working on The Xtra Factor and it's been going well," he said. "But it hasn't inspired me to do more presenting. There was no way I was ever going to let the music suffer."
And he once again dismissed rumours of a Caroline Flack romance, saying: "People think there's something going on between me and Caroline but we actually just buzz off each other. I'd love to give you guys the juicy information but there is nothing going on."