The American musician left the show a week into the series under mysterious circumstances.
A spokesperson for Channel 5 confirmed at the time that Ray J departed the series but did not reveal why he had gone, saying that details about Ray's exit would be announced on Bit On The Side.
However host Rylan Clark-Neal only said on the spin-off show: "Ray J has left the Celebrity Big Brother house. I know, it’s sad, it’s sad.”
He later told ITV's This Morning: "There's not much I can say, and we might never know the ins and outs."
Since Ray J's exit, a story by website TMZ has claimed that Ray J was taken from the house to hospital because of "extreme tooth pain".
Ray J claims that show bosses told him he wasn't allowed to return to the house after leaving for hospital treatment, despite apparently being out for just 45 minutes.
Speaking to TMZ, Ray said: "I'm ready to go to work, I'm ready to pull right back into the Big Brother house and be like 'What up, can I get back in?'"
Now Ray J's manager has revealed plans to sue over what went down behind the scens.
“We are deciding to sue because they are taking a position that is breaching the contract we agreed to," he told the Daily Mail. “They are saying due to health issues, which don’t exist with Ray, that’s the reason they pulled the plug, which is utter nonsense.
“He’s been evaluated by his own doctors and dental team and he’s OK to work."
Ray J's manager accused: “These guys used Ray to launch the show, to get it started, to make it sound like it would be a big noise.”
And he claimed: "They are not showing what really happened with his condition.
"They will show you what they want to show you. This is not a reality show. It's a sham that's orchestrated by the producers."
A spokesman for Celebrity Big Brother commented: "We always follow strict medical protocols relying on the advice of experts with regard to the health of housemates."
Celebrity Big Brother 2017 continues nightly on Channel 5.