It's more a problem with the AI... Income and combat speeds are fast, and thus, the AI tends to be able to siege frigate rush human players very effectively early on.

The early game speed boost was introduced with Alloy.  If the host isn't using alloy (meaning he has manually opened the necessary ports on his firewall), you won't see speed boost.  Original Sins didn't have speed boost or the faster game setting.  Eventually the game will bog down as everyone gets fat fleets, but it makes a huge difference at the beginning of the game.

The HOST has to open/forward port 6112 and 6000 for TCP and UDP and disable Alloy in the Sins game settings.  The one caveat is that if the host crashes, the game will try to fail over to the next person in line to become the new host, but if he doesn't have the ports open, everyone will sit there until the failover times out, at which point it will try to fail to the next person in line until it finds someone who has port 6112 open.  Pretty much his means that your game is screwed if the host crashes or disconnects, unless all of you guys open up the port 6112.

Anyhow, this is how it should work, the game should play without the early speed boost.  I haven't messed with it in years, so it is possible something changed.