The show boss made a quip about the upcoming vote following his Golden Buzzer act Boogie Storm's performance on Saturday's live show.
"Why I think this is important for the Great British public to get behind you is that we embrace people from other galaxies," he said. "I think that we proved we have some very important decisions coming up in this country.
"Voting you as the winner of Britain's Got Talent makes a very important statement."
However, viewers hit back at Simon on social media, with one tweeting: "I'm voting remain, but even I can't believe Cowell potentially just used the dancing Stormtroopers to illustrate his views."
And according to The Sun, media watchdog Ofcom is now weighing up whether the comments break its strict rules on political impartiality in the run-up to elections.
An ITV spokesperson declined to comment on the matter, but the newspaper quotes an insider as saying: "Simon was coming up with a funny line and doesn't make political statements."
Simon is understood to be anti-Brexit, recently telling the Mirror that he'll be voting remain on the 23rd of June.
"I try to stay out of religion and politics, but my gut feeling is I don’t think at this time - because it is a tricky time - you would want to be on your own on a tiny island," he said.
However, he added: "People will make their own minds up and won’t be listening to people like me."