Agreed, I just checked and I have only given six apps this permission:
- One of them being Apple's own Airport Utility
- Two being calendar apps (I guess one usage is when auto-completeing an entry based on names in ones address book)
- Two of them being 'social media', Skype & Signal, for both of which having access to the contacts list is essential when you want to contact somebody else by starting with their name only
- An app to create a map where the locations of a group people are marked.
None of them would need to upload any data from my address book to function but Skype probably does so.