The former Xtra Factor host jumped over to the rival BBC One series after leaving/getting fired (depending on who you believe) from the ITV2 spin-off show.
But Caroline insists the move was not done out of spite or revenge.
“People have said I have done it on purpose, I have gone to the rival,” she says in tonight's Alan Carr: Chatty Man on Channel 4.
Caroline explained: “I watch The X Factor on catch up. I don’t see it like that.
" The job I was doing was no longer mine and Strictly came at the right time and it was good timing."
She added: "I had three brilliant years on X Factor but this year is really good.”
Caroline went on to talk about how "intimate" the dances had already become, as she prepares to take on the Tango with pro partner Pasha Kovalev this weekend.
She said: "It’s funny how intimate it becomes though, like quickly.”
Caroline added: “You don’t know this person and then you are doing the tango and you have your thigh between their thigh all day.”
Strictly and X Factor continue this weekend on BBC and ITV respectively.