Plus 2 and plus 3 HP traits/items on trained units

Having lots of HPs can protect your investment.  Just recently, I had units get down to 10 or even 3 out of 300 HP or more.  If HP goes to 0 you lose that unit and all its experience permanently.  But if they just hang on, you can rotate them to a protected city and since they are high level, they will get a large number of HP back per turn (level x 2).  HPs protect the investment in building and getting experience on trained units.  Defense, resistance to elements and spells, and the half physical damage trait of the undead ghost units are also parts of the survival factor, but just having plenty of HP and reasonable defense go a long way in protecting the investment of XP into your units.

I've had a army where almost each unit in it had hp values of over 500 once I believe. They killed quite alot of stuff including several gods.

Sounds like a good strategy to me. I would probably choose something like +3 defense initially when my troops have almost no armour, and probably +2 initiative and Underdog (+3 attack vs. troops of higher level). Underdog is an excellent value early game damage enhancer when you're playing on higher difficulty levels and everything is higher level than your troops. If your troops have attack six then you're increasing their attack by 50%. Later on in the game Bloodthirsty is better (+50% vs. damaged troops), as you probably have attack 15+ and your troops get to higher levels, but it costs twice as much as Underdog, and Underdog gives a bigger bonus in a lot of cases at the start.

+3 hitpoints is cheaper than +3 Defense, and increases level 1 hitpoints for most races by 3/8, which increases their survivability quite a lot, so is certainly a viable option. Personally I prefer having Ironskin and one damage increasing trait. When I get leather armour I might choose hitpoints rather than defense. When you get mail armour you need to use a trait for that, of course.

High HP units are a force to be reckoned with. But watch the wage costs!

I find that early game, cheap units with crappy staffs - but trained with the "defender" trait - cost almost nothing but can tie up the enemy while heroes/better units mop up the enemy. 

If you can afford high HP units early on, go ahead, but I rarely can afford quality troops when I am struggling to find the second and third city spot.

(Later game, monsters with the Overpower trait can still shred your high-HP companies - also, Dragons)

Actually, I just had a bunch of knights live through a dragon breath with the fire protection cloak and all the +HP items.  The knights had about 8 extra HP each and it stopped the overpower from killing them.

What else was going on in the fight? Sounds like they had support. Oh, and fire protection obviously is very useful against dragons specifically.

The cloak reduced the damage enough that I didn't lose any whole units, but most were at 10 - 20% after they were breathed on with many of them losing 3 or 4 figures out of 6.

The dragon did the fear thing and froze my whole team then it breathed fire on all the knights I had which the combat had grouped close together.  The dragon went over and was biting on the closest thing, a hoarder spider.

Before it killed the hoarder spider and before it breathed again, the knights woke up and killed it.  Their attack damage ratings before they got breathed on were like 120.

Regardless, I think if they didn't have all the +2 and +3 HP items they would have all been dead.