I should amend my first post, four displays would be nice, three would be fine too. But if there is zero improvement in performance, then its not really worth it. I have three displays working now, but my third is a bit of a kludgy setup using a Matrox USB display adapter, it works but the monitor plugged into this is a bit sluggish when I change virtual desktops.
Oh there is a fairly large performance boost. But you may not feel the boost all that much unless you game or do very intensive on card renders/ OpenCL work. The 6870 is almost twice as fast as a 5770 in something like 3DMark/ Video games. Unless you are stressing it though then both the 6870 and 5770 paint pixels the same. 6870 just has the extended display support (in Windows)
The USB adapter is the main reason things feel slow for you. Mac users are basically hacking the 6870 to work in OS X. Some things don't work, most do.
Use Cindori's "Golden Guide" under his post to find out more regarding PC cards in Mac's. If you are uncomfortable with running unsupported you should just get the 2nd 5770 and run either 1 or 2 display's off it.
What kind of performance improvements are you looking for? In what apps? That may sway things.
Honestly you may just need a bit more video memory and 2x5770's would do that fully supported by Apple and drivers.