Took my M1 11” into Best Buy today to compare to the M4. There were two of the 11” and one 13”. Color was a little warmer/yellower on the M4s. Not alarmingly so, but enough to make me unsure of how well it would fit into my photo editing workflow. I came away thinking that it was probably close enough (my work doesn’t require absolutely perfect accuracy), but at the same time it was another point in favor of hanging on to my M1 for now.
Just wanted to update this with some more pertinent observations. With my M1 having charging problems, I was somewhat forced into moving to an M4. After getting it set up, its display is indeed is noticeably warmer than the M1. Not "OMG why TF does it look so yellow?!?", but if you compare side-by-side, there's a difference. My previous impression (from looking at it in the store last year) of it being likely close enough is true. But reading through some of the replies here about "Reference Mode" and fine-tuning with numbers from a display calibrator, I decided to try to get it closer to my MBP (which the M1 was already very close to), and went down a rabbit hole.
I was able to get the M4 to match the M1 and MBP very closely by using the MBP's measured white point as the target value on the M4. However, in the process of doing that, I came to the realization (which, again, was discussed elsewhere in this thread) that the M4 is actually much closer to being "right", with a white point in the ballpark of 6500K, while the M1 and MBP's white point were closer to 7000K.
Furthermore, I realized that I had been missing a step in the display calibration procedure on my MBP. Before, I'd just start the calibration process, adjusting the brightness up/down to get to my desired luminance target (120), and then letting it measure/create the profile. But there's a series of steps that should be taken FIRST, to not only get the luminance locked down, but to also adjust the white point of the display, the details of which I found in this article:

How to Calibrate and Profile a MacBook Pro with Liquid Retina XDR (MiniLED) monitor
Calibrating and Profiling a MacBook Pro with Liquid Retina XDR (MiniLED) monitor is essential but not one of the most intuitive processes. In this guide you will learn how to do it easily.

So, with my MBP now properly calibrated, and with the M4 iPP in Reference Mode (with proper fine-tune measured and target numbers entered), the two match about as close as one could expect (and the M1, as expected, now looks a bit cooler in comparison).