Simon Cowell caused outrage on Saturday's edition of the ITV talent contest after he branded Olivia's performance as "karaoke".
After the public eliminated Ricki, Simon had the casting vote on whether Olivia or Martin Bashir should join her.
The judge chose to save Martin, believing viewers would want to hear more from him.
However, social media users instantly slammed his decision - and Ricki has since added her voice to the backlash.
Talking to The Sun, the talk show legend suggested that Simon used Olivia as a "scapegoat" to generate a reaction.
"Controversy makes good TV. But it’s unfortunate that Olivia became the scapegoat, as she is so sweet," Ricki commented.
"She should not have gone home. She is stunning, she has star quality and her voice is so strong. I feel sad for her, because she really wanted this.
Olivia - who instantly burst into tears on the live show - previously addressed the result in an Instagram post, describing it as a "punch in the gut".
"I’m grateful for the opportunity as it has completely pushed me out of my comfort zone," she wrote.
"That being said, yes my elimination came as a shock. I was confused why Simon would tell me I could win this competition and then ultimately send me home."
Meanwhile, Ricki claims some of the other contestants are so talented that they'll be signed "whether they win or not".
Simon is already offering a recording contract as a prize for the winner.
"Megan McKenna is an amazing singer-songwriter, Victoria Ekanoye is an incredible talent and Max and Harvey, of course," she said.
The X Factor: Celebrity continues on Saturday at 8.30pm on ITV, when two more stars will get the chop.