Samsung Again Caught Inflating Benchmarking Scores, Phil Schiller Calls 'Shenanigans'

[LostSoul80]

First I am in no way stating that Apple's way is a standard. not even using them as an example. But the results are for the processor alone, would they have a phone model attached to the result?

And how can one explain the two greatly different benchmark scores, using the same exact benchmark tests with a different package name (as Geekbench did to uncover the "cheating")?

Here's the article BTW:

http://arstechnica.com/gadgets/2013...rking-adjustments-inflate-scores-by-up-to-20/

The answers to your questions are right in the article.
Samsung makes available to the benchmarker all the raw power available for the purpose of benchmarking. There is no way to detect a benchmarker versus any other application, if not by checking its name. That's the strategy Samsung used.

"to identify benchmarking apps by name to boost CPU clock speeds". Sounds like they are operating at higher frequencies to me.

As said, no overclocking is involved.

I am not saying if he is correct or not. But I think his point might be (and I could be wrong) is that whoever creates a benchmark or reading that benchmark is setting some "arbitrary" criteria and applying across the board. Samsung isn't the one doing the comparison - they are showcasing their hardware in the best possible light.

That's exact. Since there aren't objective and unanimously accepted rules, different subjects are entitled to use their own criteria, and the raw maximum performance is one of them.

 

[Taz Mangus]

Sure. I understand his point, I just don't think it applies to the specifics of the situation. Smartphone benchmarks, such as Geekbench, have been traditionally used to compare the CPU performance operating at "nominal frequencies" (as defined by Taz Mangus a couple posts ago). Samsung has slipped in extra code to boost performance of the CPU when running benchmark apps to the full spec'd frequency. This grossly exaggerates the performance of the Note 3 relative to other phones. No reason to do this other than to deceive.

How long would the Note 3 run at 2.3GHz with all cores active before it had a melt down or the battery died? Is that a good representation of the capabilities of the phone?

 

[LostSoul80]

How long would the Note 3 run at 2.3GHz with all cores active before it had a melt down or the battery died? Is that a good representation of the capabilities of the phone?

That is up to discussion. Is it a better representation the maximum available power of a system, or the power that the system makes available to an application? Both are valid.
The problem of the latter is that different platforms are coded differently to handle different layers in terms of power management hierarchy.
Samsung simply chose an approach - and this is totally legitimate on their part.

 

[BaldiMac]

The answers to your questions are right in the article.
Samsung makes available to the benchmarker all the raw power available for the purpose of benchmarking. There is no way to detect a benchmarker versus any other application, if not by checking its name. That's the strategy Samsung used.

Exactly. And the reason they do that is to deceive people that are comparing benchmarks across smartphones.

As said, no overclocking is involved.

Yep. As I said, I used the wrong word. Sorry about that.

That's exact. Since there aren't objective and unanimously accepted rules, different subjects are entitled to use their own criteria, and the raw maximum performance is one of them.

Again, the ambiguity that you are trying to create does not exist. If Samsung was trying to create a new point of comparison to compare across benchmarks, they would have publicized it. Instead, they sneaked in extra code to boost their scores compared to other devices that operate at nominal freq as expected.

 

[LostSoul80]

Exactly. And the reason they do that is to deceive people that are comparing benchmarks across smartphones.

That is not the case. Apple, Samsung and other companies have never agreed on how to behave with benchmarks.
It's up to those who base on or use benchmarks to document themselves, and act accordingly. Samsung has never promised to deliver benchmarks following Apple's criteria.

Again, the ambiguity that you are trying to create does not exist.

Why do you keep citing an "ambiguity"? What exactly is ambiguous to you?

If Samsung was trying to create a new point of comparison to compare across benchmarks, they would have publicized it.

Why? Where have they promised that? Nothing forces them to publicise anything. There are no norms to follow. Apple's behaviour does not represent any rule.

Instead, they sneaked in extra code to boost their scores compared to other devices that operate at nominal freq as expected.

Again, no one has ever agreed to benchmark in a way or in another.
Samsung has chosen a criteria for benchmarks. No rules prohibit it, no obligations about it exist. They have the right to do it.

 

[thasan]

Lol good one. On a related note, is there a way to ignore people through the tapatalk app?

you are welcome. sorry, i dont know.

 

[appleii.c]

That is not the case. Apple, Samsung and other companies have never agreed on how to behave with benchmarks.
It's up to those who base on or use benchmarks to document themselves, and act accordingly. Samsung has never promised to deliver benchmarks following Apple's criteria.

Man, are you serious? or are you just messing around with us? I can't tell.

No one said it was Apple's criteria... it's like this. Lets say there is a scale to measure weight, and people get on that scale to measure weight the same way. Now before you,or some other guy... lets call him Sammy, get on the scale but stuffs his pockets with rocks to achieve a higher weight, or maybe stuff his pockets with helium balloons to make himself appear lighter... that might be construed as shenanigans. It's not illegal... hey, the scale is simply there to check weight. But it can be a bit misleading. Similar to the way a boxer will fast, or even take a sort of diuretic the day of the weigh in to fall within a weight class.

Point is, there isn't a specific written criteria to follow on how to run benchmarks, but most people will take the benchmark results at face value and assume that most apps running on X device will achieve the results posted when in fact they wont.

 

[BaldiMac]

That is not the case. Apple, Samsung and other companies have never agreed on how to behave with benchmarks.
It's up to those who base on or use benchmarks to document themselves, and act accordingly. Samsung has never promised to deliver benchmarks following Apple's criteria.

That's all just BS. There is no "Apple's criteria" or benchmark agreement between companies. The companies involved in the discussion have nothing to do with the benchmarks as far as I know.

There is simply an expectation by someone referencing benchmarks that the benchmarks were run under as similar conditions as possible.

Why do you keep citing an "ambiguity"? What exactly is ambiguous to you?

You are trying to push the idea that the expectations of those people that utilize benchmarking tools to compare devices is ambiguous. They are not.

Why? Where have they promised that? Nothing forces them to publicise anything.

Why? Fairness. Transparency. Honesty.

There are no norms to follow.

You keep saying that, but one of the most respected sites that utilize these benchmarks calls it cheating. Again, no written rules, simply expectations.

Again, no one has ever agreed to benchmark in a way or in another.
Samsung has chosen a criteria for benchmarks. No rules prohibit it, no obligations about it exist. They have the right to do it.

They absolutely have a right to do it. But the reason they are doing it is do deceive the people that use these benchmarks to compare devices.

 

[LostSoul80]

Man, are you serious? or are you just messing around with us? I can't tell.

No one said it was Apple's criteria... it's like this. Lets say there is a scale to measure weight, and people get on that scale to measure weight the same way. Now before you,or some other guy... lets call him Sammy, get on the scale but stuffs his pockets with rocks to achieve a higher weight, or maybe stuff his pockets with helium balloons to make himself appear lighter... that might be construed as shenanigans. It's not illegal... hey, the scale is simply there to check weight. But it can be a bit misleading. Similar to the way a boxer will fast, or even take a sort of diuretic the day of the weigh in to fall within a weight class.

That is one very misleading example. Benchmarking is a complex series of operations, aimed at comparing different systems.
Samsung is letting the benchmarker run by making the whole processing power available.

Point is, there isn't a specific written criteria to follow on how to run benchmarks, but most people will take the benchmark results at face value and assume that most apps running on X device will achieve the results posted when in fact they wont.

Before using any kind of information, you should make sure you know what you are reading. You can't expect what you want. No rules on benchmarks exist, and if you are willing to use benchmarks, better get informed on how they are taken.

That's all just BS. There is no "Apple's criteria" or benchmark agreement between companies.

That's what I said, there is no agreement between companies on benchmarks. What is BS?
As for "Apple's criteria", it exists, as in the different way than Samsung Apple's iOS treats benchmarking applications.

The companies involved in the discussion have nothing to do with the benchmarks as far as I know.

Really? And I thought we were talking about Samsung dealing with their benchmarks.

There is simply an expectation by someone referencing benchmarks that the benchmarks were run under as similar conditions as possible.

You can't expect benchmarks to be perfectly objective, and you can't expect systems to treat benchmarking applications in the same fashion. Where did you get the impression that every system has to treat benchmarkers in the same way?

You are trying to push the idea that the expectations of those people that utilize benchmarking tools to compare devices is ambiguous. They are not.

I'm not trying to push any idea. Benchmarks can be performed in different ways, and those who want to evaluate them have to document themselves first.

Why? Fairness. Transparency. Honesty.

That publicising how a company performs benchmarks is linked to fairness, transparency or honesty is your very own opinion, backed up by nothing.
Samsung has never claimed, as far as I know, that benchmarking applications should run like any other applications.

You keep saying that, but one of the most respected sites that utilize these benchmarks calls it cheating. Again, no written rules, simply expectations.

Unless you can show me a standard, or normative, where there is an accepted description of how benchmarks should be performed, both Samsung and Apple are behaving correctly.

They absolutely have a right to do it. But the reason they are doing it is do deceive the people that use these benchmarks to compare devices.

That, again, is your very own, malicious opinion.
Samsung can treat benchmarking applications as they want. If you want to use the resulting benchmarks, get yourself documented before using them.

 

[appleii.c]

Damn, I give up. I don't have the stamina to keep this up. You win... Those benchmarks are fair honest and balanced.

 

[BaldiMac]

Samsung can treat benchmarking applications as they want. If you want to use the resulting benchmarks, get yourself documented before using them.

There's exactly the point. Why would a reasonable person expect Samsung to design its smartphones to run a program called "Geekbench", for example, at higher speeds than any other app? And why would Samsung do that other than to deceive?

 

[LostSoul80]

There's exactly the point. Why would a reasonable person expect Samsung to design its smartphones to run a program called "Geekbench", for example, at higher speeds than any other app?

A reasonable person would not trust the benchmark results of a program called "Geekbench" without knowing how it works, what it does, and how it is dealt with.
A reasonable person does not just expect that the result of Geekbench is based on the fact that the system treats Geekbench as a regular application. Samsung has never guaranteed that, so why would you expect it?

And why would Samsung do that other than to deceive?

As I said, ask Samsung. I do not represent Samsung.
The obvious answer is that they want to boast their hardware's performance, by legitimately using benchmarks. They are not cheating, since cheating meant as violation of rules presumes the existence of rules.
There are no rules on benchmarks, just like there are no guarantees that a benchmark provides an unique, reliable and objective representation of a system.

 

[Carouser]

There are no rules on benchmarks, just like there are no guarantees that a benchmark provides an unique, reliable and objective representation of a system.

There are no guarantees, but it is a reasonable assumption. The whole reason benchmarks exist is because they are presumed to be indicators of general performance, not just measures of the benchmark test itself. Samsung has exploited this reasonable assumption. They acted deceptively and no amount of "who, me?" or "caveat emptor" changes this.

 

[BaldiMac]

A reasonable person does not just expect that the result of Geekbench is based on the fact that the system treats Geekbench as a regular application. Samsung has never guaranteed that, so why would you expect it?

Which is where your argument drops into absurdity.

Do you really believe that a reasonable person should expect the same application would run at different speeds on the same device based on the NAME of the application??!?!

The obvious answer is that they want to boast their hardware's performance, by legitimately using benchmarks.

By creating an apples to oranges comparison of nominal freq benchmarks to their own spec'd freq results. Again, an attempt to deceive the users of those benchmarks.

 

[technowar]

Damn, I give up. I don't have the stamina to keep this up. You win... Those benchmarks are fair honest and balanced.

This.

Because Samsung cheated, they are on the right track and it's not their fault why the numbers were off. But when Apple makes the same mistake, they'll be the first to declare war against the company. Sad. Very sad indeed.

 

[LostSoul80]

There are no guarantees, but it is a reasonable assumption. The whole reason benchmarks exist is because they are presumed to be indicators of general performance, not just measures of the benchmark test itself. Samsung has exploited this reasonable assumption. They acted deceptively and no amount of "who, me?" or "caveat emptor" changes this.

Who says that it is a reasonable assumption, if you have no idea how the benchmarker works or gets treated? There are no rules Samsung, or other companies, have to follow in the regards of benchmarks. You can't just expect anything you want from anyone.
If you want to use benchmarks, you can't base on assumptions.
Samsung, or any other companies, have never claimed that the use of benchmarking applications would produce results based on any criteria. They can choose their own.

Which is where your argument drops into absurdity.

Do you really believe that a reasonable person should expect the same application would run at different speeds on the same device based on the NAME of the application??!?!

Using the name of the benchmarker is the only way to detect a benchmarker. That's why Samsung use a list of names. There is nothing absurd about it.

As I said:
A reasonable person would not trust the benchmark results of a program called "Geekbench" without knowing how it works, what it does, and how it is dealt with.
A reasonable person does not just expect that the result of Geekbench is based on the fact that the system treats Geekbench as a regular application. Samsung has never guaranteed that, so why would you expect it?

By creating an apples to oranges comparison of nominal freq benchmarks to their own spec'd freq results. Again, an attempt to deceive the users of those benchmarks.

No one has ever claimed that nominal frequency must be used. Nowhere it is written, so I don't see what's the point.
Again:

That, again, is your very own, malicious opinion.
Samsung can treat benchmarking applications as they want. If you want to use the resulting benchmarks, get yourself documented before using them.
Samsung is boasting their system's performance by legitimately using benchmarks, they are not violating any rule.

 

[Taz Mangus]

That is up to discussion. Is it a better representation the maximum available power of a system, or the power that the system makes available to an application? Both are valid.
The problem of the latter is that different platforms are coded differently to handle different layers in terms of power management hierarchy.
Samsung simply chose an approach - and this is totally legitimate on their part.

All this won't make any difference any longer because the indication is that the benchmark tests are going to be stealth named.

 

[tongxinshe]

Samsung can treat benchmarking applications as they want. If you want to use the resulting benchmarks, get yourself documented before using them.
Samsung is boasting their system's performance by legitimately using benchmarks, they are not violating any rule.

It's true that there are no legal documents written about it, so it's true that Samsung did not violate any laws. But, all people with common sense about the purpose and the normal usage about benchmarks can see that Samsung violated the industrial trust, and the trust of the consumers. The way Samsung played this game, everyone can see clearly Samsung is trying to cheat the customers.

 

[DeathChill]

That, again, is your very own, malicious opinion.
Samsung can treat benchmarking applications as they want. If you want to use the resulting benchmarks, get yourself documented before using them.
Samsung is boasting their system's performance by legitimately using benchmarks, they are not violating any rule.

Futuremark actually calls out this sort of behaviour in their benchmarking rules:

The Futuremark guys have a lot of experience with manufacturers trying to game their benchmarks so they actually call out this specific type of optimization in their public rules:

"With the exception of setting mandatory parameters specific to systems with multiple GPUs, such as AMD CrossFire or NVIDIA SLI, drivers may not detect the launch of the benchmark executable and alter, replace or override any parameters or parts of the test based on the detection. Period."