I thank you all for your answer. This community is really friendly, gotta love Stardock games for that! I have to admit, I completely forgot that changing the victory condition and defense bonus would not be enough, and that I would need to change the AI behaviour. That probably makes it a little bit out of reach for me.
I will fiddle with other victory conditions though, and will experiment with different map type. I'm thinking huge map with extremely rare habitable and extreme planets, maybe it will be less hard to conquer. Glad to see I'm not the only one who cannot finish games hahaha!