There are always reports of everything new "overheating". Most of the time, it's overreaction to things that people don't fully understand.
Going all the way back to M1, every Apple Silicon MBA has had a chip which can generate far more thermal power than the MBA's passive cooling system can deal with. If you max out all CPU cores on a M1 MBA for several minutes, starting at about 30 seconds in, it will begin ramping CPU clocks down to reduce power and keep temperatures at safe levels. Once the computer is fully heat-soaked, which takes several minutes when starting from cold, M1 MBAs bottom out at about 60% of their cold-soaked performance. (In a room-temperature environment, in my personal testing several years ago.)
If that's what counts as overheating in your book, well, nothing is going to be different about the M4 MBA. Just like the M1, it is passively cooled, and if you try to run extremely high compute loads for minutes at a time, it will self-regulate to avoid overheating. Also just like the M1 MBA, at no point in time will its performance be bad, and its short term burst while cold is amazing.