It's like asking why Apple couldn't have both the iOS App Store and allow flash back in 2008. The reality is that nobody would pay for native apps if free flash apps were a thing.
How many people do you think would continue to use Apple maps if Apple allowed you to set Google maps as the default app? Yet it was necessary to give Maps the time to grow and allow Apple to break away from Google's dominance.
Sometimes, a standard takes off only with enough people pushing for it, which very often means removing some degree of choice from the user.
I don't believe that this is one of those cases. As I said, pack the Lightning earbuds with the phone. If they're so superior, everyone with an iPhone 7 will get to find that out, but it doesn't make all of their other headphones useless over night. Other companies have no problems keeping legacy technology while a new standard is introduced because they don't expect an industry standard to change instantly just because they say so.