Sam got a second chance on the ITV talent contest last month when he returned at Judges’ Houses as a replacement for Anthony Russell, who quit for “personal reasons”.
Louis Walsh then chose the 20-year-old as one of his top three Boys, alongside Lloyd Macey and Spencer Sutherland.
However some viewers were a bit suspicious about it all when cameras just happened to be hanging out in Sam's house when Louis phoned to reveal the news.
But Sam insists that he genuinely had no idea about his return and told how The X Factor team had 'fooled' him into filming the scenes.
He explained: They told me they were doing a follow up story on the engagement. A lot of people online have claimed it’s a set up. They’ve said why is he looking shocked as he obviously knows he’s going through. But I didn’t."
Sam told OK! Magazine: "They told me I was going to be speaking to Capital Radio but it was Louis Walsh. I thought somebody was going to burst through the door as it was some sort of prank. I was so happy.
"People might think it was a set up but I was that good of an actor I wouldn't have to sing."
Sam’s comeback has also proved controversial because he was originally axed at Bootcamp - meaning he skipped the Six Chair Challenge entirely.
Some fans argued that Benji Matthews, the last act Louis cut at Six Chair, should have been brought back instead.
Speaking at the weekend, Sam admitted that he “totally understands” why viewers are angry about the unusual situation.
“If I was watching it myself, I'd probably be thinking the same," he commented. "I'd be thinking, 'It was a bit unfair’.
“Obviously I've been given that opportunity and I'm not going to turn it down. I've just got to take it."
The X Factor 2017 continues this Saturday at 8:20PM and Sunday night at 7:05PM on ITV.
Guest stars will be BGT winner Tokio Myers and Rita Ora.