Jessie J confirmed last month that she wouldn't be back on the show next year, and Danny O'Donoghue followed with a similar announcement this month.
With both Jessie and Danny out, it seems that Sir Tom could be the next to announce he won't be back for a third series.
“Tom won the first series and had a strong run on the second," a source is quoted by Click Music as saying. “He’s made no secret of his frustration with what happened to Leanne and he thinks it might be time to find a new challenge.
“Music is very important to him and the show monopolises his time.
Speaking ahead of the second series earlier this year, Sir Tom said he'd only return to the show if he 'approved' of his fellow coaches.
“The four of us make a great team — one without the other wouldn't work," Sir Tom said.
He explained: “One of the things I said I need to know is who the other coaches are, because I don’t want to be sitting there with somebody that I don’t admire or particularly like… there are a few of them!"
Meanwhile, Will.i.am has refused to be drawn on whether or not he'll be back for a third series, but did call on Cheryl Cole as a possible replacement for Jessie J on the panel.