It was confirmed last month that the former Xtra Factor pair will be taking over from Dermot O'Leary this year and the duo will begin work at the weekend.
"We start filming on Sunday," Caroline has revealed. "I'm so excited but nervous, too.
"I don't know what I'm most nervous about, but I'm most excited to work with Olly.
"I always wanted to work with him again but I thought he'd go off and do his music stuff - but now it's just worked out like magic!"
It's the first time that the main X Factor show has had two hosts and Caroline confessed it may be "strange" at first.
She said: "Having two of us won't feel weird for me because we've always done it on The Xtra Factor but it might feel strange for viewers."
Meanwhile, chatting to Now magazine, Caroline revealed that she was in the dark as everyone else when it came to who this year's judges would be.
However she has her own ideas, saying: "I don't know who'll be on the panel but I hope it's someone like Snoop Dogg or LL Cool J. How cool will that be?!"
The X Factor will air on ITV in the autumn.