For GPU-heavy tasks, there are advantages to the unified memory model, but the advantages aren't really that substantial when an M1 Ultra is only barely competitive with an RTX 3070 on most benchmarks.
Thing is, all those high-end GPUs that are so much better than the M1 Ultra will happily plug into a bog-standard Xeon or Ryzen tower where they will happily run Windows or Linux software optimised for x86 and DirectX, OpenGL, OpenCL, CUDA etc. Do Apple
really want to compete with Intel/AMD on making commodity workstations if their only selling point is MacOS and a shrinking handful of applications that have no Windows or Linux equivalent?
Apple
could make a "more conventional" Xeon-like ARM CPU with loads of PCIe, external CPU support and external DDR5 RAM and it would
probably beat Xeon in terms of how many cores you can fit on a chip and performance vs. power. Trouble is, power consumption - ARM's party piece - is a big deal for laptops and SFF systems and is also becoming a big deal for high-density computing and servers. However, the Mac Pro is none of those - it's not portable, has plenty of space for cooling (and multi CPUs if you want more cores) nor is it designed to be farmed in vast numbers that cost a fortune to power and air condition. Low power is still "nice to have" but not such a killer feature - especially at the cost of being able to run x86 binaries. In any case, those big discrete non-Apple-Silicon GPUs are still going to guzzle power and belch heat.
Then there's the development costs issue - the rumoured Ultra/Extreme SoCs have the great advantage that they're
basically made by linking together the same Mx Max dies that Apple can sell in far larger quantities in MacBook Pros and Studios (and the even larger-selling Mx Pro is essentially "just" a M1 Max with a chunk of GPUs left off). An ASi "Xeon Killer" with PCIe replacing half the Thunderbolt controllers and 1TB-capable regular DDR5 controller would probably mean developing a whole new die.
The point about the M1 Max/Ultra/Extreme may not be what they
can't do - i.e. beat a high-end $$$$ GPU at OpenGL/CL - but what they
can do - i.e. eat ProRes or anything else supported by the media engine for breakfast while sitting in a small-form-factor housing.
So I think I'd still put my money on a Mac Pro either being little more than a rackmount Studio Ultra
or some sort of scalable system taking multiple MPX-like Ultra/Extreme "compute" cards. It seems like better use of the existing technology. They could always keep the Intel Mac Pro alive for a few more years.