There is an
entire forum thread on Mac Pro desktops, the vast majority of which is dedicated to updating cMPs. The content has been described as overwhelming, which you should interpret: "thoroughly researched." I think for a few hundred dollars, you can make some real improvements. Consider the following:
- X5680 CPUs are less than US$50/each. These are 6c 3.33GHz.
- 32 GB 1333 Memory is less than $US100
- RX580 is less than $200, and somewhat less for used.
- NVMe SSDs are really priced abut the same as SATA SSD. About $120 for 1TB. As with SATA SSDs, you can spend more for improved reliability and better IOPs. A simple PCIe adapter is $15-30. This delivers 1400+MBytes/sec.
At the top of the first page are several sticky threads. The only thing not really covered is what I recommend doing first: netkas firmware upgrade to get to 5,1.
From what I understand of your workflow, after doing Netkas I would pop in a CPU. Note that for 2009 models, dual CPU upgrades require more skill. But single is pretty straightforward. Google: Mac Pro Technician Guide.
Then upgrade to OS to a version that works for you. I strongly recommend keeping at least 2 different OS's available. Sometimes when you go forward it's not easy to go back. Have a good backup. Upgrading to the latest Mojave brings firmware updates that unlock NVME SSD and full speed PCIe bus (GPU!).
Then upgrade to SSD. Even if you don't go NVMe, any SSD will make a dramatic improvement.
Finally, ask any relevant questions in these other threads. People are super helpful and friendly.
PS: But you really have to ask yourself this question: Should I spend US$500 to update a machine that has at best a few more years left in it? I should point out that while there is no Apple support, there is a very active thread of people who have Catalina running, and have enabled H264 and H265 en/decode in AMD GPUs. For video, this is significant, and promotes the idea that this system can go another few years. But, you could also put this toward a new MBP, which you can configure with an even better GPU/CPU. It's a significant discount, if you will. And, your cMP is 11 years old. Way past its design spec for things like power supplies, etc. As we say in the States...Just sayin'.