How do you delete old saved games?

Just finished doing that.
I went to C:\Documents and Settings\[player-name]\Local Settings\Application Data\Ironclad Games\Sins of a Solar Empire

There are a number of sub-folders where you'll find *.stats and *.save games to be deleted

Look in the various sub-folders
AutoRecord-SinglePlayer
AutoSave-MultiPlayer
Save-MultiPlayer
Save-SinglePlayer

It's a strange location for save-games but it has to do with Games For Windows certification.

It's actually easier if you go into the Options in-game, and the File Manager. Lets you delete saves, replays from within the game