Just say “hey Siri, volume 10%” when nothing is playing and it should lower Siri voice volume.
That is not a permanent set volume for Siri voice though, so if I know I want to do/test Siri stuff and it’s late at night or early in the morning I say the above command before doing anything else.
Oh and pick your volume level....10% is low enough for me.