You can turn off dictation as well. Other than that, while dictation is active (or perhaps turn on Siri and then get it to come up) lower the volume then.
I use dictation quite often. My volume is on the lowest setting but it still blasts out. The fact the phone is on silent should silence that button as all other sounds are silenced.