The problem is that Facebook has much more sensitive information in it than your iPhone address book card. Not only that but significantly MORE information in it.
It's difficult to compare the address book on your iPhone to that of your entire friends list, including possible relationships (family, friends, peers), your phone number(s), your friends phone number(s), their family members, my website address for both personal and professional uses, email addresses for personal and professional use, birthdates, favorites (books, music, movies), etc.
Think of how that could be used.
My iPhone address book card has my emails, address, phone numbers. While those are important it doesn't tell you nearly as much about me that could be used in extremely bad ways. Think more than spam and sales calls.