At the same time, for ground invasions you send massive parts of your population away as soldiers, and on the defending side everyone from the newborn to the oldest grandpa seems to join the military to fight back.
Given that population more or less directly translates to production capacity, it's much more likely that a planet's population is the number of working-age adult citizens (a class of people which mostly overlaps with military-age adult citizens) on the planet than to the total number of people upon the planet. The population growth is also linear rather than exponential or exponential modified by population pressure and available resources, suggesting that something unnatural, say spaceport throughput, is the primary factor involved in population growth rather than natural births and deaths (yes, there's the 'birthing subsidies' project, and the population growth line of the tech tree has techs which are given names and descriptions indicating that the population growth should be a function of natural births and deaths, but the actual growth rate really doesn't reflect that).
For that matter, it's arguably the case that a planet's population represents the total number of government employees on the planet rather than the planet's actual population. After all, planetary population is one of the main determinants of the output of factories, research labs, markets, et cetera which were built by the state and are maintained by the state. The factories furthermore provide the state with production at what is generally a significantly lower cost per point of production than the rush purchase costs would suggest, even if you build in a fudge factor for the extra expense of doing something quickly (okay, fine, there are conditions where a point of manufacturing may 'cost' a number of credits equivalent to or perhaps even greater than the rush cost, but that's a relatively exceptional circumstance requiring the wealth multiplier to be 10 or more times the manufacturing multiplier). And finally, there's the wheel in the govern planet screen (or the empire-wide one, if you'd rather use that), which looks like a spending allocation chart more than anything else.
Fine, we may easily say one colony ship represents not only a single vessel (or does it?), but a number of them colonizing the target planet in one larger wave.
If you accept the ship dimensions given within the game and you assume that the population numbers are in the billions, then a single colony ship cannot fit the number of people that it is supposed to carry, and the options for reducing space requirements for carrying the billions of people a colony ship is supposed to carry are limited. The time is far too short for it to be practical for the ship to reduce space by carrying embryos, infants, or small children, as they'll take too long to meaningfully contribute to the colony's production and yet a colony with a population of 3 and a given set of output multipliers has the same output whether it was founded this turn or 20 turns ago or 2000 turns ago.
The same type of military force would be your opponent on the defending side, so you have to defeat the enemy military, but not every single guy of another species living on that particular planet. You could even get mixed populations after an invasion, though that would add another layer....
How do you know there isn't a mixed population on the planet? GCIII doesn't track population by species, and the indications I see suggest population is more in line with the number of working-age or military-age citizens or even government employees than with total planetary population.