Well, knock on wood, but over the past 24-48 hours, it sure seems to me like Siri's responsiveness has improved. I haven't waited more than 2-3 seconds for any of the dozen or so queries I've made. That's a big improvement over the 15-20+ seconds I was having to wait for most responses in the past.
I guess either Apple increased server/bandwidth capacity or, perhaps more likely, the novelty has warn off among many iPhone 5 users allowing Siri to be more attentive for the rest of us. I guess I'll have a week or two to enjoy the responsiveness of Siri until the iPad Mini hits the shelves. 🙄