Mine has done that since the first beta. Very annoying. I guess my bug reports have not helped.
The new phone feature is great for iOS8 but once awhile I find that my iPad would not stop ringing even after I had answer the call on my iPhone. Same thing for Yosemite once awhile.
This just happened to me this morning. Got a phone call on my iPhone while I was at work, and I got a call from home that my iPad would not stop ringing. Turned off the iPad until I could get home to look a it.
Hopefully they fixed this with B5, we'll find out I guess.