Destroy them:
For a migrant fleet, or a race of pirates or a corporation that use outposts, asteroid dwellers etc, its colony ships, or starbases/shipyards, or asteroids would be 'it'. Once they are gone, its gone. It'd be nice if they could move however their territory in the case of the migrant fleet.
In game terms I guess when the production, research and economic production of the race is 0, the game could declare them dead.
How do they win. - Well I think this idea works strongest when they are alongside the existing empires, but either competing with them directly or existing beside them would work.
Examples of their own victory objectives:
Pirate or Criminal Race - Control, Have an Outpost on 75% of the worlds or systems in the galaxy.
Migrant Fleet - The Rebirth, Have a population equal to at least 75% of the worlds or systems in the galaxy
Pirate or Criminal - Underworld Kings, have all remaining empires paying you tribute.
Migrant Fleet - Rebuild your race or Destroy your Conquerors, Find and resettle your homeworlds, Reach a dominant military and tech standpoint
Corporate - Economic Dominance, also known as have all the empires of the galaxy in your pocket, paying you tribute.
All - Be the last surviving corporation or criminal enterprise in the galaxy, other races and factions will want to stop you dominating the galaxy in this way.
All of these races or groups follow a theme, so its really designing 1 new race type with small tweaks where it adds to the concept. You could have them all in direct competition with each other as well.
Assuming they don't have their own victory objectives, which would make more sense, and make all of this below not needed:
How does a race that doesn't use planets win with the existing systems, that is an excellent question. Well that would depend on the type of race.
Diplomatic, tech, and ascension could be achieved in the same way, so they are unaffected.
Influence victories wouldn't be possible with this type of race, or perhaps they would and nothing would change. They still can't build standard colony ships, but they can influence colonies in the same way, or the planet joins an 'independent' race, like the pirates are now. I'd just grey influence victory out when this race is in play.
Conquest victories could follow the same rule, or simply remove the colony, or even set it to 'independent' of anyone, the independent union of worlds if you wanted a fancy name. Why have an independent setting, well this allows the pirate or migrant race to keep doing business in the same way while completely removing all empires as per standard gameplay.