I had them both on for a while with no ill effect. My "All Contacts" group had many duplicates. Looks like it was the iPhone linking the contacts that have a their name field filled out but not linking the ones without names (business name only). So there were dupes on the iPhone, but not at Google or in the individual Exchange or CardDAV synced groups.
Another thing...when you're syncing only with CardDAV the "Groups" screen becomes unavailable as there really is no other group to see.
Also, groups created at Google still do not sync to iOS.
Along with exchange it's a terrifically reliable, stable solution. Syncs across iOS, OS X, Windows & Android perfectly. Every computer & smartphone I own, are instantly updated no matter what I'm using.
You can even get a free exchange account & simple detailed directions from Google. It just doesn't get any easier than that.