Many of the decisions that Apple makes in terms of product features reflect future roadmaps. When discussing such matters, it helps to think about how Apple may be setting the stage for the future. Else, it ends up like the headphone jack saga all over again, with everyone whining about how they are no longer able to use their wired headphones, and nobody trying to analyse the "why" behind such a move.
If a major change is included in one iPhone model, it is best to think of that change in context of the next few iPhone reiterations. Another way of thinking about this move is that Apple often needs to sow the seeds with one iPhone model in order to introduce future features and capabilities in subsequent iPhones.
Personally, I don't think it's would be a big deal for Apple to move from lightning to USB-C. They took the criticism when they moved from 30-pin to lightning, they can do it all over again. The reason why Apple hasn't done so, I suspect, is because it simply doesn't fit in with their long term product roadmap.
Many of the benefits of USB-C over lighting have been stated over and over again (such as faster charge speeds and faster data transfer rates), and I agree with most of them. However, I believe the reason for Apple continuing to stick with lightning is likely because Apple ultimately believes in a wireless future. They flat out stated as such during one of their keynotes even (can't recall which one offhand).
Assuming Apple's long-term goal is to eventually remove all ports from its devices, it makes sense for Apple to just stick it out with Lightning and forgo USB-C until they are ready to drop the port altogether. Switching to a new iPhone port now or sometime in the near future for "better data transfer" pretty much goes against this wireless future.
As to why Apple would do such a thing, I feel that it's not so much that Apple may come up with a wireless solution that is strictly superior to wired, but that they see wireless technology improve to the point where wires consequently have less value. For example, if you are using wireless charging to charge your phone overnight, the value of a lightning port drops because it isn't used as often.