Makes me wonder if your contacts are trying to sync and there is a problem with one of them so it's continuously trying. I had this issue a year or so ago so I re-setup my contacts for syncing and it went away.
I believe I just turned off contact syncing for the account that I was syncing contacts on. Which for me was Google. Then I turned it back on. When I went back into Phone, it took a bit to sync everything on but once it did, no more spinning problems.