Nowhere near as annoying as people who don't describe the issue they're having but instead expect us to watch a youtube video.
As far as the actual issue is concerned, I don't go swiping back furiously twenty times in the space of two seconds, so I haven't noticed it and have no issue with it. If I want to go back, I go back one or maybe two pages. If I want to go back further, I hold down the back button to see the recent history and click the page I want (if you're swiping that fast, how do you know when you've reached the right page anyway?).
If this really is how you want to use the swipe back gesture, do as BasicGreatGuy suggested and send a bug report to Apple. Be sure to actually describe the issue though, because they won't be watching youtube videos.