The former finalist replaced Dermot on the show last year but has no desire to do that again
Rather, Olly hopes to make a comeback at judges' houses after previously helping out in 2013.
“I’d love to do it again [Judges’ Houses]! I did it with Gary [Barlow] a few years ago and it was amazing," Olly said. “I’ll always be open to give my advice to contestants, I’ve always said that to Simon, if any of the contestants want some advice on the show I’ll always be there as a mentor."
Olly, who fronted the 2015 series with Caroline Flack, continued: “I felt like that was one thing that I really brought to the show last year, I was always someone there for the contestants to speak to if they needed any advice or help.
"So, obviously if Simon or one of the judges invited me to do Judges’ Houses, I’d love to lend my support.”
Chatting to the Daily Express newspaper, Olly admitted it was always going to be hard for him and Caroline last year stepping into Dermot's shoes.
"Dermot had his own way of presenting the show. It was almost Dermot’s show," Olly said. “Me and Caroline went into it trying to do our own thing, but sometimes they work and sometimes they don’t.
"I think if I was to do TV again I’d want to do my own show and put my own stamp on it.
“It’s hard when you take over a programme not just presented by a presenter, but they’re a quality presenter.”
The X Factor continues on Saturday and Sunday nights on ITV.