I have a 2018 Mac mini i7 64GB Ram 512GB SSD. I had the same 4K display. When connected via HDMI the wake time from monitor sleep was 7-9 seconds. Really slow, but not as slow as yours. I did not allow the mini to sleep, only the monitor, that may be the difference. This performance was very disappointing to me as I was used to the essentially instant on of my old 2013 iMac.
I also used USB-C to Display Port, and preferred this connection as it performed better waking from sleep--around 5 seconds.
I then got an M1 Mini and used the same monitor and connections. There was essentially no difference in the wake time of the monitor.
I have now replaced the monitor with an LG 27MD5KL-B monitor. The monitor is connected with a Thunderbolt cable. I use the default for the display resolution (retina scaled to 2560x1440). This configuration is much faster. I use my Apple Watch to login, and from hitting a key to seeing the display is less than 2 seconds.
So, you can probably solve (mostly) your problem by simply paying about $1200 for a new monitor
BTW, I love the M1 mini. It is really quick, runs very cool to the touch and is mostly completely silent--had the fans come on a couple of times when encoding video. I will never use an Intel machine again. I will likely upgrade to the next gen iMac with Apple Silicon and a retina screen, however. Particularly if the first bunch of them has second generation Apple Silicon.