Sidebar Sidebar
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

Nearly All Mobile Device Makers Cheat on Benchmarks, Except Apple and Motorola

MacRumors

MacRumors

macrumors bot
Original poster
Apr 12, 2001
68,545
39,400



Following Tuesday's report that Samsung artificially inflates its benchmarking scores, well-respected hardware review site AnandTech has published evidence suggesting nearly all mobile manufacturers, with the exception of Apple and Motorola, use CPU/GPU optimizations to game benchmark tests.

Samsung and other OEMs use a variety of methods to enhance device performance when a benchmark is detected. For example, with its Galaxy S 4 Samsung raised its thermal limits (and max GPU frequency) to get an edge on certain benchmarks and also raised its CPU voltage/frequency to its highest state when a benchmark was sensed, a tactic engaged by multiple manufacturers like LG, HTC, and ASUS as well.

In the table below, Anandtech highlights devices that detect benchmarks and immediately respond with max CPU frequency.

anandtechbenchmarks.jpg
With the exception of Apple and Motorola, literally every single OEM we've worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It's possible that older Motorola devices might've done the same thing, but none of the newer devices we have on hand exhibited the behavior. It's a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.
Click to expand...
AnandTech notes that it's a continual "cat and mouse" game discovering which devices have optimized for which benchmarks, because targeted benchmarks must be avoided.
The only realistic solution is to continue to evolve the suite ahead of those optimizing for it. The more attention you draw to certain benchmarks, the more likely they are to be gamed. We constantly play this game of cat and mouse on the PC side, it's just more frustrating in mobile since there aren't many good benchmarks to begin with. [...]

There's no single solution here, but rather a multi-faceted approach to make sure we're ahead of the curve. We need to continue to rev our test suite to stay ahead of any aggressive OEM optimizations, we need to petition the OEMs to stop this madness, we need to work with the benchmark vendors to detect and disable optimizations as they happen and avoid benchmarks that are easily gamed.
Click to expand...
Despite all of the effort that OEMs put into benchmark optimizations, the gains are negligible. The impact on CPU tests revealed a 0 to 5 percent performance increase, and a less than 10 percent increase on GPU benchmarks.

Article Link: Nearly All Mobile Device Makers Cheat on Benchmarks, Except Apple and Motorola
 
Article Link
https://www.macrumors.com/2013/10/02/nearly-all-mobile-device-makers-cheat-on-benchmarks-except-apple-and-motorola/
Also important to note that Nvidia doesn't. They don't because they've had a decade of being criticized (like AMD) for optimizing drivers to graphics benchmarks.
 
Clearly this is all about *battery life*. So, what benchmark would these phones manage if they ran "normally"? I guess the gap to iPhone5S is truly remarkable then.
 
Im Sorry but Samsung got Ys across the board. They are also the only one on that table cheating Geekbench which is one of the most popular benchmarking apps used.
 
Mad respect to Anandtech for making this chart. I love the title of the table :D

Props to Apple and Motorola for not being so cheap.
 
Last edited:
Now I'm curious to Moto and Apple if they were to cheat, how much higher the specs would be? Is it still 0-5%, or is it more? Hmmmm
 
So this thing basically looks for an executable and speeds up when it detects it. Like malware that looks for a certain executable and blocks it from running.
 
Plenty of Ars commenters suggested smartphone battery life be measured using these processor benchmarks instead of web browsing or watching a movie. It a great idea.
 
And this is why Android phones have been able to claim great performance via benchmarks and specs while iPhones have performed just as well or better in practice. I remember folks telling me that the Galaxy S2 performed better than my iPhone 3GS according to benchmarks, but in practice (side-by-side) my iPhone 3GS seemed to be twice as fast with no stutter on tasks that seemed to stutter and lag on the GS2.
 
NutsNGum said:
Guy 1: My phone is faster than your phone!

Guy 2: I have a girlfriend.
Click to expand...

But y'know, she's not here right now, so its just as 2 guys.
We can do whatever we like pretty much, and none need ever know...
 
nezr said:
This weeks lesson: don't buy Samsung
Click to expand...

No, it's more like: Don't look at benchmarks. Wait, why are you looking at benchmarks? Ok, you're looking at benchmarks. That's fine. Look, these two devices use the same processor but get different scores. Doesn't that seem funny? Wait, why are you still buying it? Yes, it will stop playing video when you look away to reflect on the fact you hate yourself and only feel fulfilled with a bigger number. It comes in gold now too! WEEEEEEE!
 
Apparently with the Note 3 and the new Note 10.1", Samsung has taken fudging the numbers to a whole new level.
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Top Bottom
Back