Westmere 12 core 2.66 memory configuration

[HDFan]

I am trying to figure out how to upgrade the memory on my 12 core 2.66 Macpro. The system was configured with 8 GB of RAM, 4 x 2 GB. I want to go to at least 24 GB, but can't figure out whether to go with 4 x 4 GB or 2 x 8 GB, or something else. Issues:

1. Various threads mention that memory should be added 3 DIMMs at a time for optimal (but possibly not real world) performance. Since I have 4 DIMMs now, I'm not sure if this means I'm already taking the minor performance hit so this is not an issue?

2. Since 8 GB DIMMS are available, I was comparing Crucial and macramdirect. I'm more interested in support, guaranteed compatibility and a lifetime warranty rather than a lower price. Crucials 8 GB DIMMs are registered, so they say that you have to remove any non-registered memory. I'm assuming the apple provided memory is non-registered and therefore would have to be removed. Since I have a support contract I will be keeping the original memory so that seems to eliminate the crucial 8 GB DIMMS? When other vendors offer the 8 GBs, will they all be registered?

macramdirect which offers only up to 4 GB DIMMS for now says "MacPro firmware prefers not mixing 4 GB with other modules". Their price for Apple Factory original 2 x 4 GB is $400, or $800 for 16 GB. I'm leaning in this direction since my configuration that way would be 100% apple.

Any advice would be appreciated.

 

[HDFan]

Just chatted with Crucial and macramdirect. macramdirect will be adding unregistered 8 GB DIMMs to their website this weekend for $349. They are not adding registered ones as they feel that the price is too high. According to them I should go with:

slots 1,2 5 with 8 GB DIMMs, 3,4,6 and 7 with 2 GB DIMMs

I should NOT just add 1 or 2 8 GB - always add at least 3 of the same type. Having 4 2 GB according to Crucial and macramdirect does not disable triple channel. Macramdirect's warning about the firmware is not correct - it's a carry-over from the Nehelam.

Crucial representative was not as helpful as macramdirect. Neither was able to tell me absolutely if my current DIMMs are registered, but we are assuming that they are not. Crucial says the only way to tell (since the google search they did with my profile information was unsuccessful) would be to call Apple.

 

[rdru]

The best places to buy ram for Apple are OtherWorldComputing and TransIntl.

They do extensive tests to their memory and OWC Memory is Backed by a 30 Day
Money-Back Guarantee and a Lifetime Advance Replacement. Warranty/Guarantee.

I used OWC memory for years in several different Macs without a single problem.

OWC states that their 8GB modules can not be mixed with modules of different
sizes. TransIntl does not state any restrictions.

3x8GB kits are $1088 and $894 from OWC and TransIntl respectively.

If mixing of 8GB and 2GB is allowed, it is better to have 3 of each size so
all your ram will be accessed in triple channel.

 

[trankdart]

Just chatted with Crucial and macramdirect. macramdirect will be adding unregistered 8 GB DIMMs to their website this weekend for $349. They are not adding registered ones as they feel that the price is too high. According to them I should go with:

slots 1,2 5 with 8 GB DIMMs, 3,4,6 and 7 with 2 GB DIMMs

I should NOT just add 1 or 2 8 GB - always add at least 3 of the same type.

Having 3 8GB sticks doesn't give you triple-channel performance unless you put them all on the same side (i.e. in slots 1-2-3 or 5-6-7).

What it's easy to forget with an 8 or 12-core Mac Pro is the Nehalem/Wetmere's so-called "NUMA" or Non-Uniform Memory Architecture. Without going into a lot of mumbo jumbo, each of the machine's two CPU's has only four local memory sockets that it can address at maximum performance (i.e. triple channel if you put 3 sticks of the same size in the front 3 sockets).

The four sockets on the other side, that is, the other CPU's local memory, are a different story. Both CPU's can of course access all the memory in the machine, but not with the same level of performance...that's why they call it "non-uniform".

If a core on CPU 1 needs data from a memory module local to CPU 2, it can't get that data at the same speed as its own local sockets, because its memory hardware can't address those "remote" sockets directly. Instead, it has to first send CPU 2 a message over the interprocessor QuickPath link, and then CPU 2 gets the data from its local memory and passes it back to CPU 1 over the same QuickPath link. In other words, there's extra latency if one CPU has to access another CPU's local memory.

The operating system is supposed to try to minimize that, i.e. to keep a program and its data together on one CPU and its local memory. It's not always possible on a busy machine, but it usually is possible, so NUMA is yet another issue that sounds scarier in theory than it is in practice.

But the thing that should be kept in mind is the locality of memory sockets.

If you're mixing module sizes, then simply having 3 8Gb modules in the computer does not at all guarantee you triple-channel memory performance unless you put them all on the same side. On the contrary, if you get three, and you put two on one side and only one on the other, what you're actually doing is guaranteeing a slowdown in LOCAL access from double to single-channel for the solitary 8GB stick. Note this is not strictly a NUMA issue because it's the same effect as putting one 8GB stick and two 2GB sticks into a single-CPU quad or hex MP. The solitary 8GB stick will get single-channel access speed.

So the configurations that the Crucial guy gave you seem weird to me. Three sticks good, yes. Splitting them up with two on one side and one on the other, bad. If you're going to split them and use different sizes, you should get four 8GB sticks, not three. That way, all your memory will get two-channel access (none will get single- or triple- channel).

Better yet, just forget your troubles and get six 8GB sticks and donate the 2GB sticks to the needy.

 

[HDFan]

Thanks. The split recommendation came from macramdirect. Didn't make a lot of sense.

So I'll go with:

1,2,3 8 GB DIMMs with slot 4 empty
5,6,7,8 2 GB DIMMS

and I assume that that should give triple channel performance since the DIMMs on each side are the all of the same type.

 

[trankdart]

Thanks. The split recommendation came from macramdirect. Didn't make a lot of sense.

So I'll go with:

1,2,3 8 GB DIMMs with slot 4 empty
5,6,7,8 2 GB DIMMS

and I assume that that should give triple channel performance since the DIMMs on each side are the all of the same type.

Yes, that will work and give you 3-channel performance on one of your CPU's (the one with the 3 8's) and dual channel on the one with the 4 2's. (it's a long story discussed on other threads...once you fill the fourth socket, the memory performance is limited to dual-channel even if the sticks are all the same).

The problem is that the imbalance (24GB local to one CPU and only 8GB local to the other) will to some extent tie the OS's hands if your machine ever gets very busy. I assume that the reason people buy double-CPU machines is that they frequently need a lot of cores and threads and programs running at once...after all, if they don't, then why not just stick with a single CPU?

However that means that you want ample memory local to both CPU's when that high activity level happens. But because of the 24/8 memory imbalance you're thinking about, the OS may have to start executing threads on a CPU which has some or all of their data on the other side, in which case you take an inter-processor performance hit.

This is all nebulous and very dependent on exactly how you use your computer. But personally, I would opt for balancing the memory sizes in the two four-socket sets. I really would consider getting that fourth 8GB stick and going 8-8-2-2 on both sides. You'll get dual-channel instead of triple-channel memory performance. But you're much less likely to get the operating system scratching its head about what to put where if and when your system gets very busy.

I obviously have no numbers to back this up but I don't believe you'll notice a difference between dual- and triple-channel memory performance in real-world situations. I'd rather have dual-channel memory performance all the time than a frequent risk of cross-CPU memory access.

Once again, it's impossible to know how much the effect of either layout will be without actually measuring your real-world workload. But I myself would definitely go for balanced memory size on each CPU.

 

[HDFan]

Just an update, as my 3 8 GB sticks arrived from macramdirect. The bottom line is that the system will not boot (flashing power light) if I mix the 8 GB and 2 GB DIMMs. I tried putting doing 10 gb on each side, 8 GB on 1,5 and 2 GB on 2,6 and the system will not boot. I tried the configurations mentioned above it will not boot. The only way it will boot with the mixed DIMMs if if the 2 GB is placed in slot 4 or 8 and the 8 GB in slots 1,2,5 (or 1,5 just using 2 DIMMS) but then of course they aren't seen at all.

I tried putting 2 8 GB in slots 1,2 and 1 8 GB in slot 5 but the Memory Slot Utility complained. It wants the 8 GB DIMMs in slots 1,2,3.

So we'll see what macram direct says, since their recommended configuration does not work.

 

[HDFan]

Wow, macramdirect has already gotten back to me. Here's what they say:

You cannot mix the 8GB modules with any others in your 12 core MacPro. The new Mac Pros are very fussy with the higher density modules and as stated on our MacPro page, you cannot mix them.
We do have an 8GB module design that will mix but the cost is nearly 50% higher which makes them way too expensive in todays world.

Load it up with your new 8GB modules and pass those 2GB chips along to your little brother...

- hope that's not a heat breaker for you..

 

[WardC]

Your optimal configuration for maximum performance will be 24GB (6 x 4GB chips) configured as 3 chips in each of the two processor's slots, with the slot closest to the processor being left empty. This allows for full DDR3 triple channel memory acceleration, and you will get the most performance out of this configuration. Your system will be slower with 16GB (8x2GB) or 32GB (8x4GB) installed. Even with 64GB, you system would be slower than with 24GB. 24GB is the optimal configuration for the dual processor 2009/2010 Mac Pro. I have a Single Quad 2009 Nehalem model, and I just removed one of my 4GB chips, giving me 12GB total, and my Geekbench score improved 112 points, with improvements in processor and floating point scores. My whole machine feels a ton faster now, no lag at all.