I've experimented with this, and I think farms are BY FAR the most important improvement, because population directly equals Raw Production (times any bonuses).
My standard build-out is like this:
1. One factory (and only one) next to the capitol or on a manufacturing bonus square if there is one.
2. Faction building (e.g., Missionary Center for benevolent civs).
3. Then spam farms like crazy, ideally having a hospital surrounded by farms. And concentrate on farm and growth upgrades in the colonization tech tree.
4. If I have special bonus tiles, I might build another factory, or a market, or an entertainment center, or a research center. But the large majority of my planets have just one factory and no research center. You don't need those.
5. You'll also want to devote an occasional tile to one-shot buildings like the Hyperion Shrinker, Technological Capitol, etc.
I routinely have planets with populations around 100.
