Once again the show has faced criticisms after it was revealed a number of this year's acts had been invited to audition.
However Ashley, who beat Susan Boyle to victory in the 2009 series, says that the tabloid stories are blown out of proportion.
Explaining how Diversity were themselves 'scouted' for the show, he told how a bloke came up to the group after spotting them performing at a shopping centre and suggested they try out.
Ashley said: "Technically, Diversity were scouted to be on the show.
"I think that it’s one of those things that depends on your definition of scouting. Did we go to the pre-audition and get headhunted, no, but the producer had happened to see us perform somewhere and said to us, ‘why don’t you enter?’"
Ashley continued: "I think if people are asked to come on, it’s not the end of the world. What makes it potentially not right is if people are groomed and helped by the people behind the scenes – that would be different.
"But I think people who were asked to come on the show and show their talent, I don’t see anything wrong with that."
Speaking to Now magazine, Ashley went on to reveal how the invitation gave the group the "push" to put their names forward.
He said: "It took that person to say 'why don’t you just come and enter the show.' And it was just a producer on their day off, walking through a shopping centre!"
Britain's Got Talent continues Saturday nights on ITV whereas Ashley is currently fronting new BBC One show Can't Touch This over on BBC One the same night.