For some reason the audio of Siri on the AirPod is insanely loud. 50% of the speaker or something like that.
Can't seem to turn it down, by either changing the Ringer volume or the Music volume. I've tried what you said, and press the volume down button while she is speaking, but it's unchanged.