Yeah I've noticed that. To add to that, today I tried calling my father, which we have a complicated Polish last name. On voice commands (on iPhone 4) it would understand it, and sometimes Siri has understood it. Today it was all over the map saying she can't find Jason hi Katie.
Sometimes I'll use my ipad 3, it works great, second later switch to iphone 5 and Siri crawls.
I think it's because iOS 6 let ipad 3 owners have Siri, and millions bought the iphone 5 which has Siri, and apple is just getting slammed. I expect it to get worse when the iPod touch 5G starts making its way into people's hands.
It was like this last year for about a month after I got the 4S though so hopefully it'll get better again over time :/
Siri is server based. You speak, it is sent to the server, who when it is your turn tries to interpret what you are wanting to do and then send back the proper information. If you have poor signal or the server is busy touching the ICON only takes a second!
I dont mind when Siri craps out on fetching a web related request. But what gets me is when its a local command like "Read my messages", "Add Applesauce to shopping list" or "Open Notes". Its very inefficient to have all commands get sent out rather than having the basic ones handled internally.