As yet nothing about the panel is confirmed although it's looking more and more likely that Louis Walsh really has quit.
Asked who they would want to see behind the judges' desk this year, Caroline replied: “Tina Turner! She’d be great. It would be good to have a legend. Or Steven Tyler, he’s amazing on American Idol. He’s my dream man. To get him would be amazing. Or can you imagine Dave Grohl? That would be great.”
Olly quipped: “I’ll do it. I’m going to have to do two jobs. I’ll say, ‘Olly, what do you think?’, then I’ll run to the desk and say, ‘You know what…?’”
Both Caroline and Olly echoed Louis' thoughts that Robbie Williams would be the man to get on the show.
Caroline said: “I’d love Robbie. He’d be the dream. He loves the show. I just don’t know if would do it. He’s been brilliant at Judges Houses."
Olly revealed: “I’ve been trying to persuade Robbie. We just did the O2 together and I was in his ear the whole time trying to persuade him. I’m failing, though.
"You need someone who has a great charisma and is very funny and lightens the audience. When I watch BGT, I like David. If I could pick anyone, I’d have Keith Lemon”
And on that note...
The X Factor will return to ITV in the autumn with an announcement about this year's judges to come "in due course".
Browse pictures of some of the rumoured names below...