I've been running through the same problems while trying to modify the game recently and I wanted to bump this post up to see if anyone had found out a way to make custom heightmaps work. I can make a background image for a planet, but the continents can't show up no matter what I try.
The heightmap I've made myself is located here.
And below would be a picture of what shows up in game so far :
From what I understand, the .dds files located in C:\Program Files (x86)\Steam\steamapps\common\Galactic Civilizations III\Gfx\planets\custom_heightmaps are not as useful as the atlas.dxatlas located there. Unfortunately, I can't seem to find a way to modify it.
Anything new on that front from more experienced modders or people acquainted with the game creators?