Apple's reliable benchmark reputation

[acj]

I have been constantly agrivated by the marketing tactics by every company, and Apple is no exception.



this article (http://www.haxial.com/spls-soapbox/apple-powermac-G5/) basically sums up and proves that Apple tweaked (or by my definition, cheated) its G5 benchmarks. It is well written and educated, and quotes many sources including the official SPEC benchmark page, which disagrees quite significantly with Apple/Veritest’s benchmarks.

Brief summary in case you don’t want to follow the link:


Apple/Veritest modified the “industry standard” SPEC benchmarks to take advantage of the G5, and crippled the PC’s by not using SSE2 and disabling Hyperthreading when it would have helped more than 50% in some benchmarks.

The numbers for Dual Xeon 3.06 vs. Dual G5 2.0 vs. Pentium 4 3.06:
SPECint_base2000:

Xeon (veritest): 889, Xeon actual: 1152, G5: 800, P4: 1085

SPECfp_base2000:
Xeon (veritest): 646, Xeon actual: 1053, G5: 840, P4: 1092

SPECint_rate_base2000:
Xeon (veritest): 16.7, Xeon actual: 21.7, G5: 17.2, P4: 12.6

SPECint_rate_base2000:
Xeon (veritest): 11.1, Xeon actual: 15.7, G5: 15.7, P4 12.7

To summarize, the current dual Xeon actually beats (equals in one case) the almost-released G5 in every SPEC benchmark, the one that Apple displays first and foremost. The P4 system, that can be had for around $1000 similarly configured, looses out in only the throughput tests likely due to its single bus.

A couple other points:

When will these REALLY ship? Will this be like the PB17’s? Why can’t Apple say, “We are announcing the G5, and we have enough built for everyone.” Remember the Dell they are comparing it too is out and not all that new.

Real world tests really matter. Darn those Photoshop tests! What 45 filters did they use? Photoshop has about 100 filters, and most people only use a few. Why aren’t they more specific? I would seriously bet the filters were hand picked to favor the G5.

Also, why does Apple screw people who just purchased a system by introducing something much faster AND somewhat cheaper. I'd hate to be the guy who just bought an ultimate G4. Or the guy who bought a 23" LCD the day before they dropped $1500 in price! The PC upgrade path is fairly steady and predictable.

I am a hardware guy, thus not so impressed by OSX vs. XP arguments. I find myself quite capable of being equally productive and efficient on OSX and XP, despite the fact I find them both to be a bloated pig of an OS.

Despite all this, I still like the new G5’s. They are priced fairly, nicely designed, and they have the other obvious advantages of Apple. On top of that, it’s easier to find optimized Altivec programs than optimized SSE2 programs (considering ratio of optimized to not-optimized).

Sorry for the long post.

[zarathustra]

while I see how people might have a point about Apple "fixing" the benchmarks, you have to consider that the higher marks for the x86 machines was achieved by using an intel optimized compiler... so, did the x86ers fix stuff as well?

Apple ran linux on the machines with a GCC compiler. Isn't that more comparable to what configuration the G5s were?

[Sun Baked]

Just read Hannibal's WWDC Wrap-up (http://arstechnica.com/wankerdesk/03q2/wwdc/wwdc.html) and stop worrying about the benchmarks.

He wrote the write-up on the 970 processors, and had some of his artwork lifted by Apple. But about the benchmarks he said it best...emphasis added...

I feel the need to say a bit about the SPEC benchmarks, because the little bake-off that Jobs did was pure showmanship. I won't go into detail dissecting those benchmarks here, but I will note that, as has been pointed out in the News Forum thread attached to the WWDC post, those were not by any means the best dual-Xeon SPEC scores that could've been used for the PC system in question. With modern computer architectures, compiler writers and chip architects no longer work in isolation on a new design, so the quality of the compilers available for a platform is every bit as important to the overall performance picture as is the hardware itself. (This issue is analogous to the issue of 3D graphics chipset performance and available driver quality--you always test with the latest and the best available drivers.) Intel's compilers are justifiably famous and widely used, and anyone who's interested in really comparing two platforms in a fair and impartial manner would have used the best compiler for each platform, would've selected the right compiler options, etc.. But of course there's nothing fair and impartial about a keynote bakeoff, so unlike some folks I didn't get too bent out of shape by Jobs's blatant stacking of the deck. I just thought to myself, "this is a keynote address, and anyone who makes a purchasing decision based on these 'benchmarks' deserves to get taken for a few grand."

The Adobe, Emagic and Mathematica benchmark bakeoffs were a bit more believable and relevant. I wouldn't be surprised at all if the dual-G5 really did out-do the P4* on these apps, because they're very bandwidth intensive and they give the G5's excellent frontside bus and memory subsystem a chance to shine. What makes a media workstation is bandwidth, and not just raw CPU power. In fact, high-end Unix workstations from SGI and Sun have always been beefier on the bandwidth side than on the CPU horsepower side--this is one reason why they're so expensive. The G5 tower looks like a real contender when it comes to moving data around, and that's what counts for pro applications.

[iJon]

already a thread on this. i think the best thing is is that apple has a future here. what, we jumped almost 900mhz in fsb and 500 mhz in processor speed and every hip happening tech that is out. even a damn spidf port, which is great. i dont see what people are complaining about. this is great for apple, even if they dont beat the pc's we are back in the game. i noticed many pc users are takin a second look at these before they make there comments. now i wanna see this against the new amd opteron and see where we stand.

iJon