Most likely, yeah
But no of course not, it sucks when that happens.
Back on my old 2014 iMac I sometimes had a "fun" situation where optimised media would lag during editing, but the harder-to-edit codecs straight from the camera would be fine; And that was because, while the optimised clips required way less CPU/GPU power to render and edit, they took up a lot more drive space, and I was editing off a spinning hard drive so the time it took to load in the footage from the drive was actually bottlenecking it. The smaller file sizes of the original files took up way more CPU/GPU but was faster to stream in from disk. - At least that's what I assume was the case based on measurements and the experience.
Do you know anyone else doing similar editing with the same media types as you?
Maybe it'd be possible for you to create a tiny mini library that just contains the section that struggles on your machine or a replica with free stock footage or something that also stutters with similar compositing or something and send to me? I'd like to try and see with my new 2020 iMac with the Radeon 5700XT how much VRAM it would allocate, since it's allocating 100% of your VRAM