You need to make multiple fleets and have them all just one move away from the planet. That way you can attack with one then another fleet (all same turn) till you win.

If you do not want to do it tahat way then there is influence flipping and diplomacy.

They wont come out with your fleets nearby.

You should not get too far behind the AI on logistics, but docked fleets have the advantage regardless, until full logistics technology is researched.

How are you on your fleet setup? The best arrangement is a couple of escorts with high defenses and a couple of capital ships with high attack. Also it is better if all your ships are equal hull size otherwise the smaller ships start the battle in a stupidly advanced position and get picked off before your larger ships can catch up

The simple way to fix ship category exploits is to have a more diverse targeting priority setup in AI fleets, or even a bit of a random element in their targeting, that would stir things up with how the player designs their fleets!