IIRC the power matrix gives 4-6 (depending on tech specialization) adjacency bonuses to each adjacent mfg, tech, or wealth building. Furthermore, those adjacent mfg/tech/wealth buildings will give 1 bonus to each other on top of providing +10% to +50% base mfg/tech/wealth.
On the other hand, those other race population buildings are worthless except for that incidental +30%/+30%/+30%.
Thus, if you specialize your planets, it is better to surround your matrix with buildings of that specialty than to have 1-2 dead zones.
If you don't specialize your planets, then you aren't minmaxing. Just enjoy the game!