Hi. I think you need to be using the Bedtime mode of the Clock app for that feature to appear.
I’m not on the beta, but just this morning the feature worked on 14.4.1. I woke up a few minutes before the alarm was scheduled to go off, had a quick look at some messages that came through during the night, then the “Turn Off Alarm“ box came on screen.