Is this on an iPhone 4s like you list in your profile? It shouldn't do that at all....
Are you AT&T or Verizon? Personally I'm on AT&T and I don't think I've ever had ending a call be an issue like that since the original iPhone... and even then I never had issues until iOS 2 when the hardware was pushed a little harder..(I think... it's been a while..).
Maybe try restoring it if it's a consistent problem? If there's no change there take it to Apple..