What is the earliest reference to "space aliens"?

Technically speaking, aren't gods extraterrestrial too? Especially the ones that were supposed to create the earth, they must have come from somewhere. And they hang out in alternative planes (Asgard) or on top of stuff (Olympus). The ones with animal heads or multiple arms are pretty alien.

Satires have the problem that they are meant to be a little silly, not convincing. I always hold out for the first scifi story being Kepler's Somnium which was an attempt to draw a convincing inhabited moon. (In circulation as a manuscript by 1611.)

The earliest not-sacred non-humans might be the scorpion-men in The Epic of Gilgamesh. (2100 BC)

(This answer isn't very historically rigorous, as I wrote it thinking I was on Worldbuilding.Stackexchange where citations and rigorous detailing aren't expected, but I think it is worthwhile here because it adds a different perspective which I would say is valid overall, and seems needed to me to relate the question to historical cosmologies instead of only looking from modern cosmology.)

I think the nature of the shift in relationship towards the universe is slightly different than you describe. One very rough but interesting perspective is that pre-empire cultures (before pantheons which had a dominant head god such as the sun god the Aryans had showed up) tended to relate to the world rather inclusively. We were part of the whole universe and everything in it was all a big cycle of death and rebirth.

The religious, ethnic and imperial conflicts that then arose, and the Aryans and other sun god worshippers, tended instead to cast their male sun god as superior to others, and humans as separate from the world and from spirit. This shift carried through into Judeo-Christianity and Islam, where we have these religious texts telling us what to believe, what's right and wrong, about how separate we are from spirit and how death is permanent etc., and a lot of that basic separateness of identity carried on into scientific thinking.

So people before all that history would tend to see an actual space alien as just part of the universe they hadn't met before, and would be interested but wouldn't necessarily think of it as other than of their universe and akin to themselves somehow.

Even in, say, ancient Egyptian or bronze/iron-age Greco-Roman cosmology (and probably many other pre-"Enlightenment" cultures), the underworld and the planets and so on were semi-metaphorical and unfamiliar from direct experience, but weren't conceived of as weird bizarre ooh from outer-space kind of things, because there was no such distinction in their cosmologies. They had plenty of imagination of other worlds and dimensions and the creatures in them. The ancient Egyptian underworld was far out, complex, and full of all sorts of interesting creatures etc.

The concept of "space aliens" as we now use it requires a cosmology akin to what we have now, and the expectations of strangeness or sameness that go along with it, which means you need to go to like 18th Century Europe at least before that's going to make any sense. Even so, it takes a long time before popular thinking or even sci fi writers are getting very "alien" in their thinking. e.g. Star Trek and Doctor Who are still full of "space aliens" who are Earth-like humanoids behaving more like Westerners than anything else.

So I think it's pretty hard to define and accept or reject any story as being space alien or not without some definitions, so it becomes fairly arbitrary depending on what you really want to know.

Cosmic pluralism (many inhabited worlds) is an idea which goes back to the Ancient Greeks. Also this is one of the beliefs that folk wisdom has that Giordarno Bruno was condemned and burned for (1600).

I would say, Voltaire, Micromegas (1752). It describes a creature from one of the planets orbiting Sirius, who visits Solar system, befriends some guy from Saturn, and they both visit Earth.

Of course it is always difficult to prove that something was the first, so let someone give an earlier reference.

I believe Cyrano de Bergerac's Comical History of the States and Empires of the Moon and The States and Empires of the Sum, both published posthumously in 1657 and 1662 respectively, precede both Swift's Gulliver's Travels (published 1726, amended 1735) and Voltaire's Micromegas of 1752.

Arthur C. Clarke credits Cyrano' books with:

being the first example of a rocket-powered space flight, and for inventing the ramjet

In the story:

Cyrano narrates the book in the first person, as if relating his own travels. In it, he uses "firecrackers" to propel himself to the moon, where he meets inhabitants who have four legs, musical voices, and amazing weapons that cook game for a meal while it's being shot.