The American musician left the show on Tuesday night, a week into the series.
A spokesperson for Channel 5 confirmed that Ray J departed the series but did not revealed why he had gone, saying that details about Ray's exit would be announced on Bit On The Side.
However host Rylan Clark-Neal only said on the spin-off show: "Ray J has left the Celebrity Big Brother house. I know, it’s sad, it’s sad.”
Since Ray J's exit, a story by website TMZ has claimed that Ray J was taken from the house to hospital because of "extreme tooth pain".
Ray J claims that show bosses told him he wasn't allowed to return to the house after leaving for hospital treatment, despite apparently being out for just 45 minutes.
Speaking to TMZ yesterday, Ray said: "I'm ready to go to work, I'm ready to pull right back into the Big Brother house and be like 'What up, can I get back in?'"
He then reflected: "It's all love, you win some and lose some."
According to the showbiz website, when it comes to Ray's fee, Big Brother "will pay Ray the balance due" with a pay out "in the high 6 figures". He had been rumoured to be getting $1 million for the entire series.
Meanwhile, Rylan appeared on ITV's This Morning yesterday but didn't have a lot more to say.
He couldn't give more details about why Ray J left Celebrity Big Brother
"There's not much I can say, and we might never know the ins and outs," Rylan said.
Celebrity Big Brother 2017 continues nightly on Channel 5.