But he says it always stays strictly professional - and he'd never date a contestant.
Olly said: "I am naturally cheeky, flirty guy. I can't stop that! I would never step over the mark."
Fellow coach Will.i.am joked: "If he had all boys on his team, I would be worried, but if there's a hot female singer, then she's going to fail because he'll be distracted!"
"I'm fine. Just because I'm cheeky and flirty, it doesn't mean I'm not doing my job," Olly insisted.
The former X Factor finalist explained: "There is obviously some attractive talent, but I have to be strictly professional as a coach.
"I've had a bit of a flirt or two on the show, but I can't change the way I am. If a woman comes on that I find attractive I might give her a compliment, but I would never be unprofessional and date a contestant. It's the Essex boy in me. You can't take it out!"
This weekend sees the blind auditions continue on The Voice as Olly meets Jessica Rabbit look-a-like Ivy
And there's instant chemistry between the pair.
He said: “She was like Jessica Rabbit. I can’t change who I am. There were attractive girls and I was flirting a little bit.”
Meanwhile, Olly has revealed the truth about a rumoured 'naked' run in with fellow coach Jennifer Hudson behind the scenes.
He said: "When we came back from the Battle stage, I walked into a room and everyone kept talking about this naked thing, but I wasn't naked!
"I had no top on because it was getting steamed and I just quickly grabbed a girl's jacket that didn't fit and threw it on to cover myself."
Olly added: "Jennifer found it funny and so the next day she gave me a robe as a present and said 'Next time you can't find something to wear, stick this bathrobe on'."
The Voice UK 2018 airs this Saturday night on ITV.