Right now I think swarm is fine as is.
There are a few ways to offset swarm.
Shield bash can break a swarm line.
An axe can deal a ton of damage to a swarm line.
Also simply,don't get swarmed, lol. Seriously form a battle line and advance like that. The guys on the flanks of the line are always going to be in the most dangerous position. Nature of the beast.