[MacBook] The Mother of all Benchmarks

[Abulia]

With my shiny new MacBook in hand (2.0 GHz) I thought it’d be fun to put the machine through its paces and see just what it can do compared to Windows and with some additional upgrades to maximize its performance.

I ran these tests over the course of a week and it took a lot of time. Feedback or criticism is welcomed and if you have an idea on how to improve, I’m interested as well. I intend to do a follow up that focuses solely on MacBook gaming performance.

Xbench
With only a handful of testing suites available for the Mac, Xbench has received a fair amount of press. This was my first time using Xbench, version 1.2.

Each test was run three times and the results averaged to yield the typical result for that configuration.

Xbench, however, appears to have several problems that cause one to question its validity. First, the program has not been updated in sometime and there have (apparently) been some compiler changes in OS X in the interim.

The results of the tests in some cases, in particular the User Interface Test, vary wildly, so much so that they can skew an overall test result by a large margin. The entire Disk Test is suspect, as slower 5400rpm drives consistently outperform 7200rpm drives, for example.

In short, I recommend using the Xbench numbers as a comparison point but to treat their results with suspicion.

Cinebench
Developed by Maxon, Cinebench is a graphics rendering benchmark program available for both Mac and Windows. This makes it handy to draw cross-platform comparisons on the same machine. Version 9.5 was used.

Each test was run thee times and the results averaged to yield the typical result for that configuration. Cinebench results have a very small degree of variance and remain consistent; they appear to be an accurate benchmark of CPU and GPU rendering capabilities. Cinebench does not measure disk or I/O performance at all.

Photoshop Test
This test is taken from the Retouchartists website and was run in Adobe Photoshop CS2, version 9.0.1. Again, because of the cross-platform nature of Photoshop this test can be run on both OS X and Windows.

Testing was measured once from the clicking of the “Start” dialog box until the “Stop” dialog box. The human margin for error is no worse than 1 second in either direction.

The test performs a number of graphic effects, resizing, and filters. This test appears to be a good measure of CPU and memory performance.

Temperature Measurements
All temperatures were measured using the CoreDuoTemp program, available online. The deep idle temperature was after 20 minutes of no activity, sleep mode disabled. The idle temperature was with periodic use, checking mail, etc, but nothing that really strained the CPU at all. The peak temperature was derived after putting a load on both cores using the standard “yes > /dev/null” command in Terminal twice (once for each core). The peak temperature was the highest temperature observed at any point during testing. The sustained load temperature was the highest average temperature noted, with the machine running under full load for at least 10 minutes.

I couldn’t find a Windows XP program that would measure temperature for the MacBook, or an easy way to put both cores under a full load.

The Results
In doing these tests I identified a number of baseline assumptions that I wanted to address to determine the “real world” results from. We’ve been told that more RAM, for example, increases OS X performance; I wanted to see by how much.

OS X RAM Performance
The MacBooks (and most Macs) ship with a paltry 512MB of RAM. In this day and age, that’s a lackluster amount by anyone’s standards. Much to Apple’s credit, OS X still runs very well on 512MB of RAM. However, what gains can be seen by increasing the amount of available system memory?

As evidenced in the chart, overall system performance increased by over 17% when the amount of memory was increased to 1GB. Photoshop rendering time was cut in half, no doubt an indicator of how memory-hungry Rosetta is, especially with Photoshop. Cinebench saw no meaningful gains in adding additional RAM. Four seconds were shaved off of boot time.

Increasing the RAM further to 1.25GB saw an approximate 10% increase in overall performance while our Photoshop rendering times shaved off another minute-and-a-half. Cinebench, again, saw no performance improvement.

Going to 2GB of RAM, according to Xbench, yielded a performance decrease of 10%, thanks to the ever-fluctuating User Interface Test. Boot time shaved off a few more seconds and another 20 seconds was removed from the Photoshop rendering test. Cinebench saw a slight improvement in rendering speed, no doubt due to dual channel memory (discussed below).

Recommendation: At a bare minimum, increase your MacBook’s RAM from 512MB to 1GB. If you intend to do any heavy graphics work or pro applications, consider more memory to make these tasks run faster, but do not expect overall system performance to increase much, if at all.

Dual Channel Memory Performance
Something we’ve been told is that using two pairs of the same memory size will allow the Macs to use dual-channel mode, increasing performance. But by how much?

Of the configurations tested, only two – 512MB and 2GB – used two like-paired memory sticks and were running in dual channel mode. Of these two configurations the Xbench Memory Test – one of the few consistent tests – did show a memory performance improvement between 5 and 10%. Overall system performance, however, was minimal, except in rendering in Cinebench: the dual-channel 2GB configuration saw a 15% in OpenGL software rendering and a 35% improvement in OpenGL hardware rendering.

Peak temperatures for the dual-channel configuration also increased by 2° C, however.

Recommendation: Dual-channel, in day-to-day use, provides little tangible or visible benefit. If you can easily and affordably configure your system with two matched pairs, do so, but don’t expect any large gains.

5400 RPM vs 7200 RPM Hard Drive Performance
This test has vexed me more than any other.

First, the Disk Test provided by Xbench gives, at best, strange results. With different drives and different memory configurations, the results are all over the map.

In nearly every case, the stock 5400 rpm drive that came with the MacBook (a Seagate ST96812AS) outperformed other user-submitted Xbench test results with 7200 rpm drives. Tapping the online database of Xbench results against “stock MacBooks” from Apple Stores, shows results that vary quite a bit. The 80GB black MacBook drive, for example, seems to perform quite well.

To add insult to injury, the Hitachi 7200 rpm drive that I received and tested (0A25016) was bested by the stock Seagate at every turn. The unit’s performance was poor, so much so that the system locked up installing OS X on the fresh drive. Disk test results were, in a word, abysmal.

The replacement Seagate drive (ST910021AS) fared better, though.

Eventually, in order to move away from Xbench’s questionable results, I had to rely on a Windows benchmarking program – Drive Speed Checker – to provide me with more accurate results. Those are included on a second chart.

All test results were run three times (sans the Hitachi) and averaged for their overall speed result.

Comparing the results of Xbench’s Disk Test and Drive Speed Checker it is clear that in 90%+ of most cases, a typical user will see little to no benefit in using a 7200 rpm drive. There may be a bit of a placebo effect in play here (“it seems snappier”) but the results do not bear this out. In fact my challenge to those who have 7200 rpm drives who claim their systems are faster is PROVE IT.

In the tests I ran the stock Seagate drive equals or beats the Seagate and Hitachi 7200 rpm drives. In only the Overall Read Speed test do the 7200 rpm drives show any clear margin of victory, but even then the difference is minimal at best.

One item that I am loathe to bring up is that perhaps there is a problem with the SATA interface of the MacBook that stunts the performance of 7200 rpm drives? Something to consider. Keep in mind that Drive Speed Checker is a Windows program, meaning that if there is a problem, it is at the hardware level and not with OS X.

Recommendation: Do not spend the additional money for a 7200 rpm drive. You will see little to no benefit. The Hitachi Travelstar drive in particular seems like a bad match for the MacBook; stick to the Seagate ST910021AS.

OS X vs Windows XP Performance
With the addition of Boot Camp it is now possible to benchmark Window applications on Mac hardware, allowing for some interesting comparisons. In the interest of time, only the 2GB configuration was tested.

In Cinebench, Windows performance was on par with OS X, but slightly worse. In OpenGL software rendering, Windows dropped by 18% but kept pace in the OpenGL hardware rendering.

The most surprising test, however, was the Photoshop test. Running a Windows XP native version of Photoshop CS2 (9.0.1), the test took 1 minute and 31 seconds longer than the same test in OS X, running the non-Universal binary version of Photoshop under Rosetta. I ran this test no less than three times with identical results.

Recommendation: There’s really no recommendation to be had here, only to dispel the myth that non-Universal Adobe applications under OS X run significantly slower than under Windows.

[Abulia]

Final Thoughts
I think the MacBook is an excellent value; the features and price just can’t be beat. The ability to have such a slick, powerful, and feature-rich system that runs both OS X and Windows is amazing. For the best performance, 1GB of RAM is necessary to be a happy MacBook owner. Eschew spending extra cash for a 7200 rpm drive or even more memory, unless your needs dictate it.

[bill4588]

i just ran that xbench program on my macbook....i got a score of 49.48 (well that's what is says next to results). is that good? what's considered a "good" score?

[plinden]

Originally Posted by bill4588

i just ran that xbench program on my macbook....i got a score of 49.48 (well that's what is says next to results). is that good? what's considered a "good" score?

I guarantee you can almost double that score by following the steps in this thread: http://forums.macrumors.com/showthread.php?t=181325

XBench is fatally flawed.

[bill4588]

well like it said in that thread, i dont really care about numbers at all. i just care about how it performs in real life, and my macbook flies to me. if there's any way to improve performance well enough to notice, let me know

[atari1356]

Wow, you did some serious testing there.

Interesting results with the hard drive speeds... thanks for the useful info.

[IJ Reilly]

Anybody else notice anything really odd in the Cinebench OpenGL scores with increasing amounts of RAM?

[Zman5225]

wow, very useful info. I especially appreciate the info on the windows and osx tests, those will help me out there.

Thanks

[dr_lha]

Originally Posted by bill4588

i just ran that xbench program on my macbook....i got a score of 49.48 (well that's what is says next to results). is that good? what's considered a "good" score?

I got around 63 with my Mac mini Core Solo, so no 49 is not a good score. I wouldn't worry about it though, Xbench scores, as others have noted, are worthless.

[SheriffParker]

It's amazing that the photoshop test in OSX with 2GB RAM beat out the windows test with the same amount of RAM. Even though the software runs naitively on Windows and not on OSX. I am really impressed. It seems that I can get my MBP before 2007 and (with 2GB RAM) run photoshop very nicely. Thanks for the info!

[steelfist]

finally some tests on ram amount vs performance. gotta get 2 gb of ram then.

[secondplace]

thanks for taking the time to run the tests. Like Bill said, as long as my macbook flies in real life, I'm pretty happy.

[IJ Reilly]

Originally Posted by steelfist

finally some tests on ram amount vs performance. gotta get 2 gb of ram then.

Actually, the Cinebench testing sort of contradicts this assumption. This is the question I raised earlier.

[Abulia]

Originally Posted by IJ Reilly

Actually, the Cinebench testing sort of contradicts this assumption. This is the question I raised earlier.

Save at 2GB, where's there's quite a jump in performance. I suspect this is in part to the dual-channel configuration at 2x1GB.

[IJ Reilly]

Originally Posted by Abulia

Save at 2GB, where's there's quite a jump in performance. I suspect this is in part to the dual-channel configuration at 2x1GB.

I suspect so as well, but that still doesn't explain the actual decline in performance with 512 -> 1 Gb -> 1.5 Gb. For those who argue that more RAM is always better, these numbers suggest that this isn't necessarily the case.

[Multimedia]

Originally Posted by Abulia

Save at 2GB, where's there's quite a jump in performance. I suspect this is in part to the dual-channel configuration at 2x1GB.

Wow! Excellent work Don. Thanks.

[Abulia]

Originally Posted by IJ Reilly

I suspect so as well, but that still doesn't explain the actual decline in performance with 512 -> 1 Gb -> 1.5 Gb. For those who argue that more RAM is always better, these numbers suggest that this isn't necessarily the case.

I suppose that depends how important Cinebench performance and OpenGL rendering is to the user.

Specifically, the person you quoted said he was going to upgrade to 2GB, which doesn't (IMO) degrade performance at all.

[IJ Reilly]

Originally Posted by Abulia

I suppose that depends how important Cinebench performance and OpenGL rendering is to the user.

Specifically, the person you quoted said he was going to upgrade to 2GB, which doesn't (IMO) degrade performance at all.

I know, but the question still remains, in my mind at least, how performance can significantly degrade with more RAM in any bench test (anything more than stock and less than the maximum, in this case). You must admit, it's not the conventional wisdom.

[uberfoto]

Very extensive.

Good job and thanks for taking the time to write it all up for us. That was a REALLY big and time consuming project that benefits the entire community!

It definitely brings up some questions about how a computer utilizes RAM and that a funky configuration, while it may be more RAM, could be less efficient than a single stick or two matched sticks. I guess the computer is doing more "running around" in order to use all of it.

I still hope CS3 runs significantly faster than running CS/2 through Rosetta...

[Abulia]

Oh, I neglected to mention that the number of History States in Photoshop were at their default settings, 20. You can get even better performance if you reduce these.

[codo]

The HD RPM section is just extremely perplexing.

Are you really suggesting that those of us that edit large video files on the go (or any sort of large file for anything, really) that’s shelling out for a 7200 has NO benefit. I've only ever purchased computers (apart from my Mini - But I don't edit on that) with 7200 HDs so I'd feel it’s a step backwards to not use one now. Maybe it is a placebo effect, who knows?!?

Fab comprehensive look at the MB there though - thanks

(I still think I'll put a 7200 in mine )

[Abulia]

Originally Posted by codo

The HD RPM section is just extremely perplexing.

Are you really suggesting that those of us that edit large video files on the go (or any sort of large file for anything, really) that’s shelling out for a 7200 has NO benefit.

No, I'm saying "in 90%+ of most cases, a typical user will see little to no benefit in using a 7200 rpm drive. There may be a bit of a placebo effect in play here (“it seems snappier”) but the results do not bear this out."

Perhaps you're one of those magical 10%.

I will say that for my use (I used the stock 5400 rpm drive a week before getting my 7200 rpm drive) that my personal experience is supported by the benchmark findings: My MacBook exhibits no superior or improved performance that is worthy of the 7200 rpm $200 price tag. Is it useless? No. I'm just saying I doubt you'll feel/see a difference.

[codo]

Very interesting indeed. I believe this summer is looking quite good for 2.5 drives so I might hold out and see.

Cheers Abulia.

[zelmo]

Excellent stuff abulia. I am currently looking to put two new boxes in our digital press area, and was thinking on the benefits of getting Intel Macs for the dual boot benefit, but was concerned that Adobe CS would have to be XP due to Rosetta. You have effectively squelched those fears.

You don't happen to have a PowerMac G5 dualie to run the same tests on, do you? I wonder how the Photoshop test compares on that hardware...

[anthonymoody]

Very interesting results - thanks for the thorough work. The HD findings make me believe there's something wrong with the SATA interface when it comes to the 7200rpm drives. I use a G4 mini, 1.42, 1 gig of ram, and have tested boot times and xbench scores with the internal 4200rpm drive and external (fw) 5400 and 7200rpm drives and found the jump to 7200 more than material (and thus worthwhile). I now use an external 7200rpm drive exclusively as my boot drive, and the internal drive in the mini as backup.

Your results are strange and interesting.

TM