Bearing in mind I don't have the DLC, and usually don't play TEC, doesn't it really come down to casualties?
If large garrisons suffer few casualties whilst defeating enemy forces while small garrisons suffer heavy casualties whilst defeating enemy forces, the faster build time may not make up for the larger number of replacements needed. Ultimately what matters is not ship build time, but total time to fully restore the garrison the system.
Your turrets and starbase(s) will also be a factor - do they soak the damage or does your garrison?