Mail sometimes takes hours to be delivered

I've set up Server 10.8 at home and (I think) correctly set up my domain registrar for my server at home. I'm using the mail service, and for the most part, it works great; I can check email perfectly with no connection errors. The problems arises when I send a test message from another non-domain account to one on my domain. Most of the time I receive the message within seconds, but sometimes, I can send a test message and not receive it for hours. It does eventually come through, but as I said, its several hours late. Any idea why this may be happening. My gut feeling is that its related to the DNS configuration on my registrar side since I honestly know very little about A and MX records and such, so I'm not even sure I'm correctly configured there.

hasenpfeffer said:

I'm not positive this is the problem, but you could look into greylisting. By default the postfix system has greylisting enabled which helps combat spam by forcing email servers to retry. The downside is at first email can take some time to get through.

Click to expand...

I whitelisted the addresses I was using for testing, and its not just incoming, outgoing mail takes hours to be delivered, if its delivered at all.