Yes, I don't think there's a way to set all devices to mute remotely.
If you want to switch back and forth, you can make a shortcut or automation to set your iPhone and watch mute on/off at the same time. But if you are just going to do this once, then it's too much trouble to create a shortcut for just that one time.