The pair - who have both won the main series - host the ITV2 companion programme alongside Joel Dommett.
Joe won I'm A Celebrity in 2008 and has been hosting the spin-off show for a decade now.
But he took to social media today that he had decided to step down in order to spend more time with his family.
Joe wrote: "I can’t believe I’m about to say this... For 10 incredible years I’ve been lucky enough to host my favourite show on telly. Being a part of the I’m a Celebrity After Show has been everything I could have dreamed of and more. But I’ve come to the decision that this year I’m going to have to leave my jungle family and stay home with my wonderful real family.
"I don’t want to miss a second with Stacey and our amazing boys so 3 months in Australia just seems too make to bare this year. I’m so grateful for everything the jungle has given me and for all of the people who have took me under their wing and given me memories to last a lifetime. I’ve made some unbelievable friends in fact family in oz and I’ll miss you all terribly."
Joe added: "@antanddec thank you boys for showing me how it’s done and keeping me on my toes I’ll be watching every night missin ya both! Well It’s not goodbye forever but just for now. Thanks for the memories @imacelebrity it’s been emotional."
Scarlett meanwhile joined Joe on the ITV2 spin-off in 2017 after winning the 2016 series of I'm a Celebrity.
She posted to Instagram: "For 3 years I have enjoyed a jungle life, being crowned queen of the jungle and having the greatest time presenting extra camp with Swashy and Joel who I now class as best friends.
"The jungle absolutely changed my life and I am massively grateful to everybody who voted for me whilst in the jungle and for your constant support to date and everybody who tuned in to extra camp 👸🏽
"Massive thank you to all the team including Richard, Angela, Toni, pecks and of course @antanddec and @hollywilloughby @joeldommett and @realjoeswashy but I have got a project that I’ve been working on (alongside my family) that I am very very excited to announce soon.
"And because of this unfortunately I will not be presenting this years extra camp but I will certainly be tuning in LONG LIVE THE JUNGLE 🕷#jungle #love #extracamp #imaceleb"
I'm A Celebrity will return in the autumn, hosted by Ant and Dec.
Details on Extra Camp - including whether or not Joel Dommett will continue - have yet to be announced.