Joe was one of two Islanders booted off the show in last week's Love Island 2019 results.
During his time on the series, Joe's ups and downs with Lucie Donlan prompted complaints from some viewers.
A number accused Joe of 'abusing', 'controlling' and 'bullying' Lucie - claims which he denies.
Appearing on spin-off show Aftersun on Sunday night, Joe said: "To think that I've been perceived in that way was heartbreaking so it wasn't nice to see.
"If I was to be [controlling] in that kind of way, I would have been kicked out. I always kept the best intentions at heart and I tried to do what was right."
Joe previously said he was "gutted" about how his and Lucie's relationship had been perceived by some viewers.
"I didn’t manipulate or abuse her," said Joe following his exit. “I am gutted that it has been perceived in that way because it wasn’t like that at all."
He insisted: “I always had Lucie’s best interests at heart and I promise you I would do everything and anything to help her because she had a hard time in there and didn’t get on with the girls. I was trying to help her and make her experience as amazing as possible. I thought I did right.”
In an interview with The Sun newspaper, Joe refused to apologise.
"I am not going to say sorry," he said. "I may have worded things wrong and the cameras are on you 24/7. I may have slipped up a bit but I am only human.
"I am young and inexperienced so maybe some things have come out differently to what they are. I just hope people can see me for who I genuinely am and what I genuinely am. I am not that person."
Suggesting Lucie would not see things the way some viewers have, Joe concluded: “If it was how people said it was I would have been removed. They do take precautions in there.”
Love Island 2019 airs nightly at 9PM on ITV2.
Monday night's latest show will see a new recoupling that will leave one girl dumped.