xlr8yourmac got some user feedback on the ATI HD 3870 regarding video editing. Basically, he says the 3870 is better than the 8800 because the nvidia drivers are unstable.
http://www.xlr8yourmac.com/Graphics/ATI_3870_Mac_PC_review/radeonhd3870_review.html#updates
"Just installed mine yesterday in our 8-core Xeon, 8GB RAM, and it blows away the stock card away for video editing tasks. (I asked what the previous card was - he later said his 2008 Mac Pro had the 8800GT card, which is a BTO option - see later comments below.-Mike) In fact, I had to disable OpenGL acceleration in After Effects (latest After Effects from CS3, v8.0.2.) and Motion (v3.0.2) in order to have them work properly, but the ATI 3870 works perfectly. I'd had to turn off real-time updating of video scopes with the previous card, or it would drop frames on playback. Works perfectly with the 3870. It's like getting a new machine.
The page about the Mac Pro (2008) graphics card shows the one we were using in the center, the black NVIDIA with the white "arrows" - GeForce 8800GT. (8800GT card is a $150 BTO option now (was $200 option originally) on the current (2008) Mac Pros.) The Mac was purchased a month ago, brand new from Apple. (And we have the hardware raid card as well, with 3 drives in a Raid 5 configuration.)
I had problems with the card from Day One. (It was about a month or so, as we had planned to get the new ATI card as soon as we could.)
...on the ATI, everything works as expected (and as it used to on my old G5).
The NVIDIA gave OpenGL errors in After Effects if I enabled OpenGL, and it would not play HDV video in the timeline of Final Cut Pro and at the same time display real-time videoscopes, without dropping frames. (Which the old G5 could handle.) Does NVIDIA simply not have good drivers? Or Apple not know how to work with them? Strange.
I'm running 10.5.4, with no Software Updates outstanding.
Hope that helps!
(He later wrote)
Still loving the new 3870 card... It really makes the 8-core machine feel like it's working at full capacity. I'm having to get a new license for Nuke and can't wait to try that out. (I love Nuke... only problem is that it was originally designed for film work and doesn't handle interlaced video as well as Motion or AE. But it's a killer program and I think it will fly on 8 cores.)
-Wayne F."