It's been all change on the ITV show this summer with a brand new family.
Olly replaced Dermot O'Leary together with Caroline Flack while the panel has seen Mel B and Louis Walsh gone and Rita Ora and Nick Grimshaw join.
There's also been a host of other improvements made in a bid to keep viewers tuning in each weekend but Olly admitted: "Everyone was unsure how it was going to fit together at the start."
He told the Daily Mail newspaper: "You’re putting three or four different people in – me and Caroline plus two new judges so you never quite know how its going be, but from day one it’s really worked and you have to give credit to Simon Cowell for seeing that and believing in it."
However so far only the auditions have been filmed with the live shows still to come which Olly admits to being nervous but prepared for.
"It’s quite nerve wracking but like anything I’ve just go to go out there and enjoy it," he told the paper. "I’ve got a great team behind me so I just want it to be a really fun show.
"From a presenting point of view I know what I can do and what I’m capable of. I don’t think there’s any pressure for me. I’m not letting people see me as a presenter. I’ve just got to go out there and have a good time."
As for the talent on show this year, Olly teaser: "You’ve got to expect the unexpected and there are girls and guys and groups who you don’t expect to be good and then they become world beaters.
"That’s the nature of the beast. I’ve been a part of it and it’s a great feeling to find a talent that you don’t expect to be talented."
The X Factor begins on ITV later this year.