emails getting stuck in the outbox

[superspiffy]

For some reason Mail has trouble with sending mails. Any emails being sent gets stuck in the outbox and will only send unless I restart Mail. When I restart Mail, the email finally gets sent but it still gets stuck in the outbox and will stay there indefinitely. So every time I restart Mail, it resends that email one more time because it's still in the outbox. I'd have to delete it manually from the outbox. What's going on here? I've setup my outgoing server alright, and there was a point when it worked fine but at some point this issue came up.