I does affect us all in some way, but it's not really an issue with iOS.
You should go ahead and ask a developer what fragmentation does. What it does is that it makes newer API less attractive to a developer, and it will require developers to keep in mind users that are still running out-to-date software in order to ensure stability and performance towards system versions and revisions otherwise unnecessary.
The sole reason why you see millions upon millions of apps update with new iOS 8 API's and features, stability fixes and performance enhancement on day one with iOS is due to the fact that developers know that the vast majority of iOS users will update within a matter of days, or weeks.
If this had not been the case, the developers wouldn't have cared. The competition within App Store is so huge, and the continuous flow of efficient iOS upgrades forces (which is a good thing) app developers to keep their apps up-to-date and enhanced with all the latest features and whatnot fast in order to keep them relevant in the marketplace.
Take Android for example, its been over two years since they released their "Holo" optimised UI and UX-design, and still less than 10% of the apps on Google Play Store have optimised towards these guidelines at all... And lots of Google's latest API's introduced in Android 4.X have yet to be utilised in the majority of apps on Google Play Store simply due to the fact that there are so many out-of-date devices out there that it makes no sense for developer to focus on the latest and greatest software, and the latest and greatest hardware because that's not where their potential demographic lies.
As a Android developer you wont develop a app or a game with the Galaxy Note 4 in mind, instead you have in mind all those millions upon millions older and lower priced models in the market as they make of a much higher market share for Android compared to the top-of-the-line modes forcing developers to ensure that their apps and game will run on both Android 2.X, as well as Android 3.X and 4.X and you will also need to make sure it runs on that slow Galaxy Young as well as that high-power Note 4 or you will lower your potential customer base quite a bit.
Why else do you think games like Balloon Tower Defence runs much slower on a Galaxy S5 compared to an old iPhone 4S? How come on level 80+ things start to lag as hell on the S5, while you can keep going until level 130+ on the old iPhone 4S before the same thing starts happening on that phone? How come graphics in games tends to look better on older iPhone devices compared to brand new, top-of-the-line Android devices?
It all comes down to optimisation and utilising the latest and greatest features, programming languages and API's of the platform. Android has a huge disadvantage compared to iOS on this, and of course people would want all iOS users to update as well in order to keep things attractive for developers in the future.
