Last month, both Caroline and Olly announced that they will be leaving the ITV talent contest after just one series fronting the main show.
Their departures came after Olly's performance was panned by viewers, with the singer suffering embarrassment after he announced that contestant Monica Michael had been eliminated - when the vote was supposed to go to Deadlock.
Now Matt has weighed in on the situation, saying it's a "shame" that Olly's slip-ups have reflected on Caroline, whom he worked with on The Xtra Factor with in 2013.
"I think it’s a real shame that Caroline’s taken a bullet when she was the only redeeming feature," the comedian told the Radio Times.
"She was ace. I think she saved Olly Murs’s a**e on more than one occasion. I think it’s a shame that she’s been made to look like she wasn’t good enough because she was brilliant."
Caroline and Olly originally presented The Xtra Factor in 2011 and 2012, with Matt taking over from Olly when he left to focus on his music career.
The pair were reunited for the main show last year as replacements for Dermot O'Leary.
Dermot is now expected to return for the upcoming thirteenth series, with boss man Simon Cowell reportedly telling him to 'name his price' after realising how popular he is with the public.
Matt can be heard hosting the drivetime show on the newly-relaunched Virgin Radio when it hits the airwaves next week, while Caroline will be back on TV later this year with a new series of ITV2's Love Island.