Four Of Diamonds are said to have considered saying no to an X Factor return.
The four piece were originally sent home at judges' houses by mentor Louis Walsh.
But Four Of Diamonds then got a second chance when they were brought back into the live shows when another of Louis' acts dropped out.
According to The Sun newspaper, the group - made up of Sophia, Caroline, Yasmin and Lauren - weren't instantly keen to come back.
A source said today: “They make no secret about wishing they had a different mentor, all of the girls wanted Nicole as she’s most experienced in their field, but unfortunately they don’t get a say.”
Apparently the girls were worried about how Louis would be as a mentor, especially given he admitted to not liking working with girl groups.
The insider claimed to The Sun newspaper: "They nearly declined the offer to come back, they knew Louis wouldn't push them - he just doesn't care.
"Sophia is the most annoyed about it, she's getting really frustrated with the way Louis talks to them, telling them they need to work harder when he's not doing much to support them."
At the weekend Four Of Diamonds were in the bottom two but Louis, Simon Cowell and Nicole Scherzinger all saved them from the axe.
"It's hard because we haven't had the time to build our fanbase in the same way, so we can't put ourselves across in the same way the other contestants can," the girls said in the week. "But we're so grateful for what we've had because it's been amazing."
The X Factor is back next Saturday and Sunday night on ITV at 8PM.
It's Halloween weekend and there will be a guest performance from reigning winner Louisa Johnson.