Oh man, this sounds very annoying. Does anyone know if this is addressed in the newest VisionOS 2.2?
I'm the same as you, assquatch(lol), I seriously hate when I have to re-set up my desktop spaces, app windows, etc.--what a workflow killer. I find MacOS is pretty good about this when connecting/disconnecting physical external displays--it typically keeps all the windows together on a new desktop Space, then moves the Space back to the external display when re-connected, which is ideal. But it's not totally consistent. Sometimes it will put a window in the wrong Space, and change the order of the Spaces. Very annoying when it happens, but still much better than how my Windows laptop handled dis/connecting displays.
As far as going back and forth between the virtual Mac display and the physical display, I think what might be a good solution is when disconnected, just fit the entire virtual display within the Mac physical display. Even if it's an ultrawide virtual display going to non-wide physical display, just show it like a long letterbox bar, and the section containing the mouse cursor would magnify as big as possible (kind of like how the cursor magnifies the Mac dock icons). From here you can either leave it until you connect to the VP again, then it will be exactly as it was, or if you want to work on the physical display, then you can move windows around and arrange them, either within the Space or move them to a different Space--and as windows at the farthest side edges of the ultrawide are removed, that section of the virtual display is removed until it goes back to the normal native size/ratio of the physical display. But it would also be nice if you could somehow save each arrangement and go back to each with a click of a button. And if any apps in the arrangement were closed, maybe it would ask if you want to open them. And if any new apps were opened, I suppose they could just default to a new desktop Space.