I like running Memtest on a new system overnight (or for a few hours at least), which can at least show if the memory system is working well.
I've seen that there's a Memtest for OS X, though I don't know how it works (or if it works on Intel Macs), since the one for normal PCs boots from a disc, totally bypassing the OS, and I think the OS X one somehow ran from inside the OS?
Should return zero errors. If there are errors, it's usually the RAM itself, though I think it's possible it could be the memory controller.