Best performance for Adobe after effects CS6?

Probably your disc caching

From what you have written and what I have read other places you might want to have your caching set up to go to the ssd. Im running Cs6 on a 2009 imac and it screams. Cant wait to get a new machine with a solid state drive to cache to.

OpenGL and CUDA are totally different things. Both are only usable for certain things, so prior to such an investment, you should ensure that it will actually benefit you.

I use a 2009 Mac Pro that I upgraded to a 6-core CPU and 5870 for After Effects heavily. I only have 32GB of RAM, but I found a couple things that helped quite a bit:

1) Setting aside 6GB for other applications. This limits AE to only 26GB, but I stopped having crashes after changing this. I also have no page outs or swap file use this way, and the most I've seen my system use with AE and Premiere open is 29GB.

2) Setting aside "4 CPUs" for other applications. The system thinks I have 12 CPUs due to hyperthreading on 6 cores, and I allow only 8 threads to run simultaneously by checking the "Render Multiple Frames Simultaneously" and keeping four aside. The reasoning is similar to setting RAM aside... prevent crashes from AE sucking up all the available RAM and CPUs.

3) 2GB RAM allocation per background CPU. I tested every combination on my 6-core system, and this one was fastest. When changing this, I could double or triple the time it took to render a comp.

4) I have an 8-disk RAID 6 tower that holds my media, previews and scratch files, along with a second volume for media cache files in a 2-disk RAID 0 internally. Providing plenty of fast space lets me play DSLR footage from a 5DII natively at full resolution without even rendering it out, and that's software-only CUDA acceleration in Premiere. (AE doesn't use CUDA as far as I know, anyway.)

This is on dual monitors (one being a 30" ACD) hooked to the 5870. I get no beachballs in RAM previews or anything. It's nothing short of wonderful.

I hope it helps!

I had a similar setup at work that was giving me great pains. It was a real let down considering the price paid. You need to follow the above recommendations, it covers all the bases. Also consider getting an SSD for the scratch disks and media caches. If you still get kernel panics, check your RAM to make sure there is no faulty stick as well. It turns out that on the setup I was using one of the sticks was bad, but we only found out much later, and that was causing kernel panics. Note also some AE plugins and effects don't like multi-frame rendering.