I find "raise to speak" very temperamental too.
Here is another thread about Siri's faults and I found an apple discussion topic elsewhere about it. We're not the only ones. Amusingly/frustratingly the answer is similar to the iPhone 4's reception problems "you're holding it wrong". Apparently we're "putting it to our ears wrong"
🙄