I had the same error and...
I also received a message that said cannot connect to caldev. Funny thing is I have never setup any account to point to a caldev address since caldev is for calendars. My contacts have been working flawlessly for the last 6 months even after an iOS 5 upgrade until this week.
I did some more checking and it looks as though the carddav.address.yahoo.com url is pointing to caldav.calendar.yahoo.com which is Yahoo's calendar server.
You can verify by entering either address in your web browser and it always comes up to the calendar page. I verified this at multiple sites to make sure it was not my isp.
I also did a tracert on both addresses and the endpoint address for both although different still point to the calendar when I enter them in a web browser.
tracert carddav.address.yahoo.com
final result: cardgate01.cal.vip.ac4.yahoo.com (216.109.126.235)
Navigate from chrome
https://216.109.126.235/
You attempted to reach 216.109.126.246, but instead you actually reached a server identifying itself as caldav.calendar.yahoo.com
tracert caldev.calendar.yahoo.com
final result: calgate01.cal.vip.ac4.yahoo.com (216.109.126.246)
navigate from chrome
https://216.109.126.246/
You attempted to reach 216.109.126.246, but instead you actually reached a server identifying itself as caldav.calendar.yahoo.com
I sent this to Yahoo and will see what response I receive.