Here's some homework for anyone who's complaining about this: Write some code that loads a few GBs of data into RAM, draw the data, and animate it from a tiny thumbnail to fullscreen. Report back on how you solved all the performance bottlenecks. Then you can critique Apple all you want!
Gigabytes of stuff in memory is not necessarily the point. If swapping works at all, it should work on an iPad Pro with 6Gb of ram with maybe just some upper limit on per-app memory that is just a little lower. (Apple will even allow 16GB of app memory on devices that only have 8Gb of ram) But here is the weird thing. Apple doesn't support virtual memory swapping on the base storage model of the M1 iPad Air because 64Gb of storage is too small, but this model seems to run Stage Manager without limitation. So, the question becomes, is app swapping really necessary for Stage Manager or is it really something else that makes it possible? Nobody is doubting there is a performance hit. Many of us are just questioning the actual reason for it and speculating that what Apple finds unacceptable, may actually be tolerable, even usable, if implemented with some constraints.