CariElf CariElf

1.1 Memory/Performance/Save Game Problems - look here

▼

1.1 Memory/Performance/Save Game Problems - look here

Check here for unofficial test exes

from GalCiv2 Forums

A lot of people seem to be having problems with memory with 1.1, so here are a few tips until we can get another build up:

1) If you haven't done it already, increase the amount of virtual memory available to Windows.

To increase the amount of Virtual Memory available to Windows:
 
1) Right click on the My Computer icon and chose properties
2) Click on the advanced tab
3) Under performance, click on the Settings button.
4) Click on the advanced tab
5) Click on the change button
6) Change the initial size to the recommended size and then increase the maximum size.
 
As a note, you should make sure that you have enough hard drive space left for the maximum virtual memory size or you'll start having problems whenever Windows has to allocate more virtual memory. You can get the size of the virtual memory in gigs by dividing the number by 1024.

2) If you still get the problems, play on Medium or smaller galaxies.  Setting planets and stars to Rare will also help, although you will probably want to set the Habitable Planets to Abundant to make sure that you do have enough good planets.  You could also try making a bee-line for larger hull sizes so you won't need as many ships.

3) Defragging your hard drive may also help if you haven't done it in awhile. 

I'm sorry that we didn't catch this earlier, but the developers and artists all have 2 GB of RAM (or more) so none of us have had this problem.   Fixing the memory issues is our top priority at the moment.

If you have Windows 2000 or XP you can try this UNOFFICIAL test exe. I will be updating this link periodically until we get the official update out.  I'm using an newer version of the DirectX SDK on my computer than the one that Brad uses to create the offical builds, and the version I'm using is not supported on Win 98/ME.

Before extracting the contents of the zip file to your GalCiv2 install folder, backup your GalCiv2.exe and Lib3D.dll and your save games, just in case. 

http://sd.stardock.com/carielf/gc2/GalCiv2MemTestWinXP6.zip

This test file is meant to

1) fix crash when loading save games

2) reduce save/load times (may only be slight difference)

3) make it less likely to crash while saving.

update:

4) This version attempts to abort saving rather than just crashing when it runs out of memory while saving.  However, it is still possible to max out your memory so badly that the game will crash even if you're not saving.

I am going to try to make it abort saving gracefully rather than crashing when it runs out of memory, because on a gigantic galaxy with everything set to abundant, you are going to max out your memory if you only have 256 MB of RAM.  This will give you the chance to try adjusting your page file in the hopes that you will be able to save again.

I am also going to try to reduce the memory usage.

Update 5/5/06: I am un-stickying this thread and making a new one to try and get more people using the test exe.

--Cari


 

50,484 views 55 replies
Reply #26 from GalCiv2 Forums Top

The problem is more likely due to memory fragmentation rather than memory leaks, because before 1.1 was released, I ran BoundsChecker and there were only a few small leaks - about 28 K - which I did fix.  I just posted a blog in the dev journals section about memory fragmentation, if you're interested. 

Reply #27 from GalCiv2 Forums Top
Any idea when this is going to be fixed? A week, a month?
Just to have an idea when I can play my new favourite game again (hoping flatter will get me everywhere)
The Political Machine 2024
Reply #28 from GalCiv2 Forums Top
Couple comments - I mean these in a constructive way, so please don't take them as a flame.

1) I am all for giving devs fast, beefy hardware on which to develop the game. You literally can't put a price tag on maximizing developer productivity by making sure they have a fast box, 2 or even 3 displays, and whatever their personal idea of an ideal working environment happens to be (quiet, or music, dim light, bright light - whatever it is).

HOWEVER, your *testers* should have more than just one of those machines; they should also have a range of hardware configs to test on - and one or more of those should exactly match the game's posted minimum specs. There's no excuse to not know *exactly* how your product runs on the minimum and recommended configurations (as well as "maxed out" hardware). I realize Stardock may not have the excess budget for a hardware lab with a hundred different configurations, but you definitely need to get a few machines with low-end specs (on the plus side, you ought to be able to get several of these for the prices of a single dev box ). A natural corrolary is to be sure to test several of the most common video card chipsets (even if you're partnered with NVidia or ATI), at a variety of performance points.

2) Memory fragmentation: At the risk of getting way too technical, is there any particular reason you're not building the game with a non-default heap manager (e.g. SmartHeap)? It may take some work to plug in, depending on how you've done your existing memory management, but you shouldn't have to reinvent the wheel to address this. You may even be able to use the memory allocators from something like ACE if you don't want to shell out the $$ for a commerical solution like SmartHeap (I don't remember the exact licensing terms for the ACE libraries, but I'm pretty sure there's at least one commercial option).

-X
Reply #29 from GalCiv2 Forums Top
Mine is the same way. I can play for any number of hours as long as there is no autosave and I don't try to save. If I do try to save it is about a 50% chance of "out of memory" crash to desktop. What is worse the crash takes the save game spot with it. So if I save on an old save spot then that spot is gone also....



I have the same problem. The chance of an out of memeory crash is about 30% for me though...

But i think the problem began with the latest beta of 1.1, at least for me!

I hope the devs can fix that soon because we cannot play anymore!
Reply #30 from GalCiv2 Forums Top

Any idea when this is going to be fixed? A week, a month?

I don't have a date yet for an official update, but we will probably be working on the memory and performance issues and testing our fixes for the rest of the week at least. 

Memory fragmentation: At the risk of getting way too technical, is there any particular reason you're not building the game with a non-default heap manager (e.g. SmartHeap)? It may take some work to plug in, depending on how you've done your existing memory management, but you shouldn't have to reinvent the wheel to address this. You may even be able to use the memory allocators from something like ACE if you don't want to shell out the $$ for a commerical solution like SmartHeap (I don't remember the exact licensing terms for the ACE libraries, but I'm pretty sure there's at least one commercial option).

We haven't used a lot of 3rd party solutions in the past because we didn't really need to, but as the scope of our projects have expanded and we've started using newer technologies, we've added things like Bink, the Miles Sound System, and the PUG xml parser.  I will look into the applications that you've mentioned, and check out CodeProjects for open source solutions when I get a chance.  For now, though, I think that making better use of our existing memory manager will suffice.

Reply #31 from GalCiv2 Forums Top
Cari, Since many of us have the existing issue I am sure others, including myself will be willing to test any update you make. At the end of each day if you make any changes you need to test, post the update and I will be willing to try it out in the evening. I can set up performance monitor to monitor the game turns and forward any results you require. At this point the game is unplayable and I can get results back to you by the time you start work in the morning. Seems some people have the game simply crash during game play and others, like me only have a crash during the save game process.

What I do wonder is because I have been playing if all my save games will be no good after the patch.... What I am asking is do you think because of the memory leak/fragmentation do you think the save games have bad data in them?

Good luck on fixing this,
Mike
Reply #32 from GalCiv2 Forums Top
I second that Themanthatisi !

For those that are having this issue, an idea:

I have found that running in window mode with the task manager open (Ctrl+Alt+Delete) to the performance tab helps keep track of when a crash can be expected. In my case I have a duel monitor system, though that should be necessary to see the task manger quickly in windowed mode.

I simply watch the value labeled 'Available' in the Physical Memory column. When this value gets close to 100, its time to save and restart GalCiv2. This makes the game playable. Just check the memory before each end turn, and if it is getting close, save and exit to windows, then run the game again.

Depending on your system, you may need to watch the value labeled 'Total' in the Commit Charge column. Just see where the game crashes, and stay under that limit.

Just something to get me by until the patch is out.
Reply #33 from GalCiv2 Forums Top
Rather than dealing with ever growing ini (or xml files) for that matter, an approach which is not scalable, why not find some small-footprint freeware db engine, and store ship & planetary info in the database. Access times will be speeded up tremendously, meaning saves and loads will operate faster, plus it is hugely more scalable. There comes a time when using flat files (ini files, xml files, etc) simply becomes too expensive. There are several good small footprint (10-20MB) db engines that would meet the need.
Reply #34 from GalCiv2 Forums Top
I'm having the same CTD issue every 4-8 saves. I have an 2 year old alienware with 1gig RAM and 256 MB video card. I've been PC gaming since the Apple II+ and this game reminds me of the old days when I used top mess around with the extended memory in my config.sys file on my IIc. Genrewise I'm coming to GalCiv2 from a MoO II background and interestsieyour game is beating the other games in my current repertoire: Oblivion, DDO(cancelled), WoW(3 60s now cancelled), Civ 4(looking for someone to play with!!) and many other though a little dated titles.

I will use the "windowed" memory monitoring solution for now and "refresh"(exit out and restart) my game every x turns. Please note that this is an incovenient time drain due to the LONG load times on saved games and possible crashes when switching between the game and the task manager-I'm 36 have a wife and 21 month old little girl so time is precious. Yes- I do play huge but the rest is normal...currently beating intelligent opponents then will move to next notch(on huge again)

Pancakes!!
Reply #35 from GalCiv2 Forums Top
Cari,

If the hulls are "expensive" (Brad's word) what could I do to limit the "cost" of them? Is it the user-added extras? Numbers of hardpoints? Would one of the empire's designs be less costly?

I am happy to limp along until you get a new version up, but would like to know what (if anyhting) I could do. Uh, besides limiting myself to smaller maps. Gotta have room to expand, ya know!

Thanks,
Jon
Reply #36 from GalCiv2 Forums Top
Actually, using larger ships would help. The more ships you have, the more memory it uses. Fewer large ships is better than lots of little ones.
Reply #37 from GalCiv2 Forums Top

It's the amount of components and jewelry on them.  If you make really plain ships with only a few pieces of jewelry, that will make the section and key names in the ships shorter. It will also mean less parsing. 

Reply #38 from GalCiv2 Forums Top
My crashes occurr only at time of saving, which usually goes on for 7 or 8 minutes before finally crashing. I had created a custom ship , I called Super Nova class. It was huge with many weapons and armor and defenses etc.
But I never used any skins or extras for appearnce. I probably had 10 to 12 of these active at any given time in my huge galaxy. I has these ships before the 1.1 beta was released with no problems . I only started to use the 1.1 beta after its fifth iteration.
Reply #39 from GalCiv2 Forums Top
Talon> you fiddled with your config.sys on your Apple IIc, eh? marvellous accomplishment, considering that Apple IIe's have never run MS/PC-DOS.

~~~

Never had a crash while saving, autosave is off. My crashes in 027 seem to be when the AI is thinking or, rarely, when a move path is being calculated (and GC2 comes up with some really bizarre, highly inefficient move paths).
Reply #40 from GalCiv2 Forums Top
2) Memory fragmentation: At the risk of getting way too technical, is there any particular reason you're not building the game with a non-default heap manager (e.g. SmartHeap)? It may take some work to plug in, depending on how you've done your existing memory management, but you shouldn't have to reinvent the wheel to address this. You may even be able to use the memory allocators from something like ACE if you don't want to shell out the $$ for a commerical solution like SmartHeap (I don't remember the exact licensing terms for the ACE libraries, but I'm pretty sure there's at least one commercial option).


@Xentax: I just have a question about what you were saying about heap managers.

I was under the impression that memory management was the job of the Operating System? What does this SmartHeap or similar products do?

I ask because having taken an Operating Systems class at University, we had to, among other things, implement a virtual memory manager for our Operating System OS/161 (BSD Unix but with most of the stuff stripped out). That way, whenever you malloced some memory the OS would handle all the work (allocation, memory fragmentation management, etc). I thought that this was the same under Windows... or perhaps this heap manager does something else?

Just interested
Reply #41 from GalCiv2 Forums Top
It's the amount of components and jewelry on them. If you make really plain ships with only a few pieces of jewelry, that will make the section and key names in the ships shorter. It will also mean less parsing.


Cari, i find the fleet processing really sluggish, removing a ship from a fleet takes up to 7 or 8 seconds.

This is on a AMD X2 2600 Mhz with 1 GB of Ram using very plain designs.

Is this something that is normal? I deleted all my old designs but guess i need to start a new game to see if this helps?
Reply #42 from GalCiv2 Forums Top

I was under the impression that memory management was the job of the Operating System? What does this SmartHeap or similar products do?

I ask because having taken an Operating Systems class at University, we had to, among other things, implement a virtual memory manager for our Operating System OS/161 (BSD Unix but with most of the stuff stripped out). That way, whenever you malloced some memory the OS would handle all the work (allocation, memory fragmentation management, etc). I thought that this was the same under Windows... or perhaps this heap manager does something else?

The OS does have a memory manager but it depends on the OS how much it actually does.  Brad is continually lamenting the lack of automatic garbage collection in Windows.  Java actually has its own garbage collector, but Java is still too slow for games like GalCiv2. 

The advantage of having your own memory manager and not relying on the OS is that you can track where memory is being allocated, pinpoint leaks, and make the most effective use of memory for your specific application.  Different applications have different memory needs and different implementions are more efficient.  It sounds like SmartHeap is a 3rd party solution that takes care of memory management for you, so that you don't have to code it yourself.

We have a memory manager that we use for certain parts of the code, but it's kind of klunky, so we've replaced a lot of his custom classes that used the memory manager with STL classes.

 

Reply #43 from GalCiv2 Forums Top
Hi CariElf,

The OS does have a memory manager but it depends on the OS how much it actually does. Brad is continually lamenting the lack of automatic garbage collection in Windows. Java actually has its own garbage collector, but Java is still too slow for games like GalCiv2.


Well, I skimmed through some of my lectures from the Operating Systems class that dealt with memory management and I am pretty confident that memory should not fragment to the point where it will actually make much difference. While, from what I read, the XP system is not that great, it should still not be noticeable. As I believe someone else said either on this thread or on the Dev Journal thread, it is most likely not the memory being fragmented. Rather, I believe that you guys might have some memory leaks that you might have not seen and that the profiler was unable to catch. Of course, take everything I say with a pinch of salt . When finals are over I will take a closer look.

Also, in regards to the garbage collection, I believe I have heard my profs mention several times that there are versions of C/C++ that do automatic garbage collection. I haven't looked into it since we are mostly doing Java now anyways.

The advantage of having your own memory manager and not relying on the OS is that you can track where memory is being allocated, pinpoint leaks, and make the most effective use of memory for your specific application. Different applications have different memory needs and different implementations are more efficient. It sounds like SmartHeap is a 3rd party solution that takes care of memory management for you, so that you don't have to code it yourself.

We have a memory manager that we use for certain parts of the code, but it's kind of klunky, so we've replaced a lot of his custom classes that used the memory manager with STL classes.


Yes I always hated that about C/C++ . The best part is when you refer to a pointer that you already freed and you get the nice little Segmentation Fault . But yeah, best to stick with the STL, as I recall in the Dev Journal you were saying that you were increasing the buffer size by one each time, hehe best to use a vector

Hehe anyways, back to studying
Reply #44 from GalCiv2 Forums Top
I've been playing my game on a laptop and the laptop is really feeling it. I increased my vitrual memory to about 4,000 as the max. I keep having to increase it. Its probably cause Im playing on a HUGE map with alot of races. I have been playing the same campian for about 3 months haha. Im really in to it ,and I want to finish it. I was wondering if there is anything else I can do to free up space. Since my laptop has official been modified just to play this game already ha.
-Dannar
Reply #45 from GalCiv2 Forums Top

Hmm...my last reply didn't seem to go through.

MysterD03, you may want to check out the test exe I linked to in the first post of this thread.

Reply #47 from GalCiv2 Forums Top
At last my CTD is cured....this is how I struck lucky

I changed 2 things, not sure wich has fixed it, but I have been playing all morning without a crash, where as before I was getting them every other move.
1 I dropped the background star density to 500 ( looks better anyway )
2 Installed cachemanXP and set "auto recover if ram goes below 200 mb" and "cpu theshold at 35%"

That was it and it plays fine now with not a hint of crashing or slowdown.
Im so happy
Reply #48 from GalCiv2 Forums Top
The fix fixed my load problem. No CTD after load. Thanks, Cari
Reply #49 from GalCiv2 Forums Top
Dyna,, you mentioned dropping Star Density to 500.. How did you do that ?
Only options I see for Density are "scattered" clusters,, tight clusters etc..
Reply #50 from GalCiv2 Forums Top
I changed those files and the save times are great.