This last week I've been struggling with re-stabilising my other PC after the PSU went bang & being sent a faulty replacement.
Doing this brought me back to a level of competance that I've not had for years by having no choice but to research all the shit I'd forgotten or had been superseded.
What I discovered is that speed of RAM is hardly noticeable & only shows a difference in benchmark tests that you wont notice yourself, but quantity of RAM is a different thing altogether.
GPU or RAM? firstly you need to know what exactly it is that youre going to be doing with the PC which you already have done.
What you are needing to do is remove any bottlenecks in your system & you can never have enough RAM as this is the usual place for PC Lag.
Vista seems to put a lot more pressure on your RAM than XP as its trying to reduce the pressure on your gfx card. If youre gaming even a pair of 8800s in SLi isnt enough to allow you maximum gfx settings on Crysis.
I personally suggest more RAM than a fancy overpriced GFX card that in 3 months will cost you 1/3rd of what they cost today.
Whatever RAM you choose be sure to check your boards support on the QVL to be 100% sure its going to run reliably & stable as not just any RAM is going to work in your board as I'm sure youre already aware.
Hope this helps just a little