Hey Siri constantly breaks

Does anyone else have the problem where Hey Siri stops working after a period of time? The only thing that works for me is to restore the watch as a new device, which is pretty painful considering this happens every few days. I was hoping the problem would be resolved with 3.1, but actually installing that update last night broke Hey Siri.

Any advise?

(My gf is having the same problem with her Series 2 as well).

 

Am I really the only one with this issue?