The tenth series winner of the ITV show spoke to Simon in the US, but feared she was just being pranked.
Sam, who triumphed in the weekend's final at Wembley Arena, said: "I didn't actually believe it was him, I thought it was an impersonator. I was like, 'Do I call him sir, boss, Simon?'
"He's just so down to earth. He was like, 'Hi darling, you're absolutely wonderful, babe'.
"I was like, 'Wow, this is Simon Cowell'. I pictured Rory Bremner as I was talking to him.
"He was really, really nice and it was inspirational to talk to him. He said that he'd been watching me, he'd seen something in me and he got high hopes for me."
Speaking at a mini press conference yesterday evening in London, Sam also spoke about her early plans for her debut record.
The 34-year-old dismissed reports earlier this year claiming that Simon wanted to 'rush' her album for release by next Spring.
"It wouldn't be a case of rushing it," Sam insisted. "I don't want to do a cheesy album with loads of covers, I want to do something a bit different.
"I've got a few ideas up my sleeve.
"Simon says he wants my input. He wants me to go in there with a few ideas and [not] just let everyone else make up my mind for me, so I've got my work cut out."
Browse pictures of Sunday night's X Factor 2013 final results below...