Generally, these Nehalem/Westmere Xeons run best in triple-channel memory configurations. I run 32 GB in mine (4x 8 GB DIMMs), but obviously since it's 4x 8 GB sticks, it's actually not running in triple-channel.
EXAMPLES of triple-channel configurations for single/dual CPU tray 5,1s:
- Single CPU with 3x 8 GB sticks (24 GB)
- Single CPU with 3x 16 GB sticks (48 GB)
- Dual CPU with 6x 8 GB sticks (48 GB)
- Dual CPU with 6x 16 GB sticks (96 GB)
Obviously you could go with 4 GB sticks, too if you wanted a small amount of RAM. Whereas you could theoretically do the following with all your RAM slots:
- Single CPU with 4x 8 GB sticks (32 GB) - this is how I have mine configured
- Single CPU with 4x 16 GB sticks (64 GB)
- Dual CPU with 8x 8 GB (64 GB)
- Dual CPU with 8x 16 GB (128 GB)
Obviously other configurations exist, but these are just the easiest examples to show. But these above configurations are NOT triple-channel because you'd be saturating all of the DIMM slots.
So to answer your question—try running either 3x 16 GB or 3x 8 GB (for 48 or 24 GB respectively), and see how it works. I should actually test that out myself, too!
EDIT: I noticed your RAM is running at 1066 MHz, so it would be advantageous for you to upgrade to a Westmere CPU so that (provided your DIMMs are 1333 MHz), you could run them at their full 1333 MHz. Plus, I noticed my system was running much snappier after I upgraded from the Nehalem garbage to a Westmere CPU (see signature).
EDIT #2: attached About This Mac screenshot
