I think that your video card is on it's limits here. For the gamer, there are three main factors that have the most influence on how much graphics RAM you'll need: resolution, visual quality detail settings, and AA. For the most part, 512MB of RAM seems sufficient to push one of these factors to the limit, and in most cases, it can handle two of them at once. But if you plan to maximize all three--the highest resolutions, visual quality settings, and AA--then more video RAM than 512MB is a good idea. The primary consideration should probably be resolution, because this is a hardware limitation for many of us. Typically, 20" or smaller monitors will cap out at 1680x1050, so in this case, spending extra on 1GB of graphics RAM(to buy new card) might be frivolous. Conversely, if your monitor is 21" or larger with a 1920x1200 native resolution, purchasing a graphics card with 1GB of RAM might be a wise investment.
Try to lower down resolution (I know it will look...hmmm a bit ugly when not native), but...
Did you tryed in Options.ini to set >
[GraphicsOptions]
..................
UseAdvancedShaders=1
.............
That should help a lot.