Contacts are generally managed through the Address Book application, and although there is some integration with Mail. Probably best to head into Address Book itself and delete whatever you want.
If you are talking about the autofill of addresses, go to Window>previous recipients and delete from there. I delete all of mine from time to time just to keep things neat. Has no affect on the Address book entries.