They probably have a military starbase near the planets. All planets in the area of effect of a military starbase cannot be invaded. This applies to core worlds and colonies. You must find and destroy the military starbase first. There should have been a message about this when you tried to invade the planet.
