It slowed my iPod down to a stuttery crawl, due to lack of RAM. Once it was up and running though, Siri was buttery smooth, so I don't think the A4 is much of a performance bottleneck.
The iPhone 4 has the same amount of RAM as the 4S, so I'd imagine that speed problems are less pronounced on the 4 than on my iPod.