Samsung Again Caught Inflating Benchmarking Scores, Phil Schiller Calls 'Shenanigans'

[samcraig]

one thing i know for sure. u are obsessed with siding samsung samcraig..on almost every occassion I have seen in this forum (maybe i see only the negative ones from u but thats my perspective)

Please look up the word obsessed. Hyperbole there my friend. But yes - clearly your perspective.

Antennagate was debunked by statistics.

But not by users. So there you go.

 

[FrankB1191]

It's safe to assume all android benchmark scores are a scam and not to be trusted.

I'd like to see Anandtech use some type of in house benchmarking suite that only they have access to these scams from happening.

If you haven't read AnandTech's article yet, they do use different tests that act as a moving target.

 

[appleii.c]

one thing i know for sure. u are obsessed with siding samsung samcraig..on almost every occassion I have seen in this forum (maybe i see only the negative ones from u but thats my perspective)

I'm sorry - are you suggesting that there are companies that aren't corrupt in one way or another

Further - can you not sense that I was speaking tongue-in-cheek/in hyperbole.

Please don't pretend to know why I post what I post or what I do or do not know about any company. You have no idea what I know or don't know.

Eyeroll indeed.

Ah HA... is that what the "Sam" represents in your name, samcraig? I knew it! (kidding)

 

[thasan]

Please look up the word obsessed. Hyperbole there my friend. But yes - clearly your perspective.



But not by users. So there you go.

No. I stand by my statement. Anyone who reads this forum regularly knows this. without specifically searching for anything, i have never ever seen anything positive you had to say about apple. i wonder why you come here. perhaps your life so so full of positives that you look for negatives... ?

im not your 'friend' as i dont see anything positive in your 'life' in this forum. sorry. in any case, have fun.

 

[1Life]

No. I stand by my statement. Anyone who reads this forum regularly knows this. without specifically searching for anything, i have never ever seen anything positive you had to say about apple. i wonder why you come here. perhaps your life so so full of positives that you look for negatives... ?

im not your 'friend' as i dont see anything positive in your 'life' in this forum. sorry. in any case, have fun.

I agree. Dude seems to enjoy posting snide remarks and defending anyone criticizing Apple. Just ignore the guy.

 

[thasan]

I agree. Dude seems to enjoy posting snide remarks and defending anyone criticizing Apple. Just ignore the guy.

i do it mate. i know its not a good idea to feed certain groups of ppl in the forum. but... well... sometime u gotta to feed them.. otherwise they might lose their part time job

 

[LostSoul80]

No, I stated you could start with common sense and a dictionary.

Then, your first statement is missing any sort of back up, since you haven't provided what you meant by "purpose of a benchmark".

In case you forgot your words, you claimed that

Most people that have an understanding of the purpose of a benchmark. Such as Ars Technica and Anandtech.

Not only you haven't provided evidence of the "Most people" reference, but also of what you meant by "purpose of a benchmark".
Would you mind clarify?

You are trying to create ambiguity where there is none. Samsung and others are cheating. If they included an extra battery that was only accessible during battery tests, that's cheating. If they including extra pixels in the screen that are only turned on for screen tests, that's cheating. If they filled the device with helium to decrease its weight when it is weighed, that's cheating. If they remove a portion of the case when it's measured, that's cheating.

You are missing the point.
Samsung is not adding imaginary RAM, nor is it using more powerful processors to conduct the benchmarks.
The results of the benchmarks come straight from the regular Samsung hardware. There is no cheating involved. The operating system allows its core to run at full potential when a benchmark application is caught running.
You are trying to create ambiguity, for all I'm concerned, by providing completely irrelevant claims.

Kaffee: Corporal, would you turn to the page in this book that says where the mess hall is, please.
Cpl. Barnes: Well, Lt. Kaffee, that's not in the book, sir.
Kaffee: You mean to say in all your time at Gitmo you've never had a meal?
Cpl. Barnes: No, sir. Three squares a day, sir.
Kaffee: I don't understand. How did you know where the mess hall was if it's not in this book?
Cpl. Barnes: Well, I guess I just followed the crowd at chow time, sir.
Kaffee: No more questions.

Unless you can show me a standard, or normative, where there is an accepted description of how benchmarks should be performed, both Samsung and Apple are behaving correctly.

Yep. That's just not purpose of benchmarking smartphones. Overclocking the processor specifically for benchmarks at a speed not available to the apps that the customer would use is cheating. It is intended to deceive the people who are comparing the relative performance of various smartphones.

You clearly have no idea what overclocking is. Samsung is not overclocking anything. Have fun reading: http://en.wikipedia.org/wiki/Overclocking


Also, what are you basing on your claim about a difference between benchmarks of computers and of smartphones?

Absolutely. But, again, that requires everyone to be aware that's what is being compared.

No one has ever claimed what was being compared if not processor performance. Again, feel free to read: http://www.primatelabs.com/

 

[samcraig]

Ah HA... is that what the "Sam" represents in your name, samcraig? I knew it! (kidding)

You figured it out. My mom and dad must have known one day that naming me Sam would have cultural relevance on the internet ha ha

No. I stand by my statement. Anyone who reads this forum regularly knows this. without specifically searching for anything, i have never ever seen anything positive you had to say about apple. i wonder why you come here. perhaps your life so so full of positives that you look for negatives... ?

im not your 'friend' as i dont see anything positive in your 'life' in this forum. sorry. in any case, have fun.

Right - you have me completely figured out. Good for you. I'm miserable and have never posted anything positive about Apple and never anything negative about Google or Samsung. And that's not hyperbole.

i do it mate. i know its not a good idea to feed certain groups of ppl in the forum. but... well... sometime u gotta to feed them.. otherwise they might lose their part time job

My checks have clearly been lost in the mail. Do you know who I can contact to get paid for my posts?

 

[s2mikey]

Some people out there do prefer Android, and Samsung's one of the best Android phone makers. Don't compare two companies that focus on two entirely different experiences.

Exactly. Apple has plenty of BS policies, strange rules, and whatever else. Everyone has to weigh those against the positives of the company and make a decision. Same for anything you buy from any company. I like iPads versus other tablets but phones are a different story. Samsung being arouind is a GOOD thing for everyone else. If Apple was the only game in town... the prices would double and upgrades would come to a halt.

I dont why Apple is seen as some charity or something.... for example, they use the same cheap, enslaved labor to make themselves huge profits. So do most tech companies. Its WRONG in all cases.

I just dont get blind, company loyalty. Unless you have a financial stake in the company. Overall, they dont care at all about you, they just want your money.

 

[applefan10466]

i do it mate. i know its not a good idea to feed certain groups of ppl in the forum. but... well... sometime u gotta to feed them.. otherwise they might lose their part time job

Lol good one. On a related note, is there a way to ignore people through the tapatalk app?

 

[BaldiMac]

Then, your first statement is missing any sort of back up, since you haven't provided what you meant by "purpose of a benchmark".

In case you forgot your words, you claimed that

Not only you haven't provided evidence of the "Most people" reference, but also of what you meant by "purpose of a benchmark".
Would you mind clarify?

Ignoring the semantic argument you seem to be focused on, I've mentioned the point of the benchmark quite clearly.

"It's to compare the relative performance of the phones for certain tasks."

Samsung was increasing the performance of the processor for those tasks ONLY for benchmarking purposes.

You are missing the point.
Samsung is not adding imaginary RAM, nor is it using more powerful processors to conduct the benchmarks.
The results of the benchmarks come straight from the regular Samsung hardware. There is no cheating involved. The operating system allows its core to run at full potential when a benchmark application is caught running.
You are trying to create ambiguity, for all I'm concerned, by providing completely irrelevant claims.

It's the same thing as my examples. The processor is limited to a specific performance level for normal operation by Samsung. However, the limit is bypassed specifically for benchmarking purposes. Customers will never see that extra performance in running non-benchmarking applications.

Unless you can show me a standard, or normative, where there is an accepted description of how benchmarks should be performed, both Samsung and Apple are behaving correctly.

You're just repeating yourself. Did you not understand the point of the quote?

You clearly have no idea what overclocking is. Samsung is not overclocking anything. Have fun reading: http://en.wikipedia.org/wiki/Overclocking

Perhaps I used the wrong word. Could you explain what the relevant difference is for the purposes of this conversation between overclocking and boosting CPU clock speeds over the normal limit?

Also, what are you basing on your claim about a difference between benchmarks of computers and of smartphones?

I didn't mention computers.

No one has ever claimed what was being compared if not processor performance. Again, feel free to read: http://www.primatelabs.com/

Again, they are comparing processor performance. No one is arguing that. However, smartphone processors are limited from their max performance in order to conserve power. Samsung bypasses those limits in this case specifically for benchmarking software. Why do you think they do that?

 

[applefan10466]

I'm amazed that people are defending Samsung here. They purposely showed results that a consumer cannot attain with the stock version of their product.

It's shady and the defense of this company is very interesting.

 

[LostSoul80]

Ignoring the semantic argument you seem to be focused on, I've mentioned the point of the benchmark quite clearly.

"It's to compare the relative performance of the phones for certain tasks."

Samsung was increasing the performance of the processor for those tasks ONLY for benchmarking purposes.

Samsung was not increasing any performance of the processor for any tasks. Samsung is allowing the processor to run at its maximum potential, which is a natural and reasonable way to provide a benchmark.

It's the same thing as my examples. The processor is limited to a specific performance level for normal operation by Samsung. However, the limit is bypassed specifically for benchmarking purposes. Customers will never see that extra performance in running non-benchmarking applications.

It is very different from your examples. Samsung is not artifically using different hardware than what customers get. Samsung is, instead, using a benchmark as a means to measure the maximum performance of its hardware, the same hardware customers buy.

Nowhere is written that the system should allow a benchmarking application to measure the performance as if it were a normal application. As I stated multiple times, anyone is free to consider benchmarks with their own perspective, and the maximum performance is as valuable as any other one.

You mention that "Customers will never see that extra performance", but you are missing that the result of a benchmark does not provide any clue to the customers about the absolute performance of a system. Benchmarks do not provide day to day usage information, they are used to compare different systems under a certain criteria, and Samsung has chosen to provide maximum system performance.

You're just repeating yourself. Did you not understand the point of the quote?

You seem to be avoiding to answer what I wrote two times now.

Perhaps I used the wrong word. Could you explain what the relevant difference is for the purposes of this conversation between overclocking and boosting CPU clock speeds over the normal limit?

Samsung allows the platform (Android) to stop managing certain aspects of how the processing power is used for the purpose of benchmarks. No overclocking is involved. Overclocking leads to superior processing power, while Samsung is using the raw processing power available to every handset.

I didn't mention computers.

I just did.

Again, they are comparing processor performance. No one is arguing that. However, smartphone processors are limited from their max performance in order to conserve power. Samsung bypasses those limits in this case specifically for benchmarking software.

Samsung chooses to use all the processing power available to run when benchmarks are running. Also, care to show a source detailing how the process for which "smartphone processors are limited from their max performance in order to conserve power" work?

As for the reason, you may as well ask Samsung. Probably it's for marketing, and they have the right to do it, since no standard of norms exist to support one way against another one for benchmarks.

 

[appleii.c]

Samsung chooses to use all the processing power available to run when benchmarks are running. ...

Correct me if I'm mistaken, i'm not a benchmarking/processor guru... but if this is the case, shouldn't the benchmark be for the "processor" itself, and not the phone? I mean, if the result is on the processor (without any battery saving limitations for the phone, and no system stuff running), then you cant tie the phone and OS to the result, no?

Also, unless I read the Arstechnica article incorrectly, which I kinda did get through it pretty quickly, the only time the CPU uses that "extra processor boost" is when accessing a specific list of benchmarking sites. To me that isn't real world results.

 

[Taz Mangus]

What a surprise Samsung denies that they were cheating on the benchmarks again: http://appleinsider.com/articles/13...-unsubstantiated-denial-of-benchmark-cheating

See they weren't cheating on the benchmark, Samsung's statement proves it.

Samsung activates various performance modes when running specific benchmarks, including activating all idle cores, increasing the clock rate and making other, unknown changes that boost its graphics performance over identical hardware.

While these changes boost scores by 20 to up to 50 percent (in AnandTech testing), they do so at the cost of overheating components and running down the battery. These impacts are so costly to the overall experience that Samsung does not activate "maximized frequencies" when running real apps, because if it did, its other benchmarks would suffer, particularly battery life and product reliability.

Normal operation, my arse.

 

[LostSoul80]

Correct me if I'm mistaken, i'm not a benchmarking/processor guru... but if this is the case, shouldn't the benchmark be for the "processor" itself, and not the phone? I mean, if the result is on the processor (without any battery saving limitations for the phone, and no system stuff running), then you cant tie the phone and OS to the result, no?

Geekbench claims to provide benchmarks for the processor alone. Seems reasonable to me that a platform, such as Android, lets benchmarking applications run at full potential, with the purpose of evaluating the system's processing power.

Processing power is managed by the operating system. That is why, when comparing hardware alone, it's important to remove the operating system's influence.

Comparing different platforms in an objective, fair way is very difficult, if possible at all.

Also, unless I read the Arstechnica article incorrectly, which I kinda did get through it pretty quickly, the only time the CPU uses that "extra processor boost" is when accessing a specific list of benchmarking sites. To me that isn't real world results.

Android may choose, depending on background applications running for example, to allow an application to work in a certain environment. There are too many variables to consider, but the processing power is used by (1) the operating system, which distributes it to (2) applications.

Two devices with the same processing power, but with different platforms, will perform differently from an application's point of view, depending on each OS's optimization - background processes, save battery settings, the platform's way to handle single windows, etc.

Samsung uses their hardware in a different way than Apple's, and this doesn't make Apple's way a standard or anything.

 

[BaldiMac]

Samsung was not increasing any performance of the processor for any tasks. Samsung is allowing the processor to run at its maximum potential, which is a natural and reasonable way to provide a benchmark.

It is very different from your examples. Samsung is not artifically using different hardware than what customers get. Samsung is, instead, using a benchmark as a means to measure the maximum performance of its hardware, the same hardware customers buy.

Nowhere is written that the system should allow a benchmarking application to measure the performance as if it were a normal application. As I stated multiple times, anyone is free to consider benchmarks with their own perspective, and the maximum performance is as valuable as any other one.

You mention that "Customers will never see that extra performance", but you are missing that the result of a benchmark does not provide any clue to the customers about the absolute performance of a system. Benchmarks do not provide day to day usage information, they are used to compare different systems under a certain criteria, and Samsung has chosen to provide maximum system performance.

You seem to be avoiding to answer what I wrote two times now.

I just did.

Samsung chooses to use all the processing power available to run when benchmarks are running. Also, care to show a source detailing how the process for which "smartphone processors are limited from their max performance in order to conserve power" work?

As for the reason, you may as well ask Samsung. Probably it's for marketing, and they have the right to do it, since no standard of norms exist to support one way against another one for benchmarks.

You don't even seem to be acknowledging what Samsung is actually doing. Let alone the purpose in comparing smartphone benchmarks. Not sure where to go from here.

Samsung allows the platform (Android) to stop managing certain aspects of how the processing power is used for the purpose of benchmarks. No overclocking is involved. Overclocking leads to superior processing power, while Samsung is using the raw processing power available to every handset.

As I suspected, you don't know what the difference is, if there is a difference at all.

 

[LostSoul80]

You don't even seem to be acknowledging what Samsung is actually doing. Let alone the purpose in comparing smartphone benchmarks. Not sure where to go from here.

Not sure either, since you are not providing any kind of answer to what I wrote.

As I suspected, you don't know what the difference is, if there is a difference at all.

Overclocking causes the processor to operate at faster frequencies. Samsung is not overclocking the processor of the Note 3 in any benchmark.
Allowing the stock processor to run at its frequency, disabling some Android's management limitations for the purpose of comparing the hardware alone, does not imply overclocking. Not sure what you're asking there.

 

[appleii.c]

Geekbench claims to provide benchmarks for the processor alone. Seems reasonable to me that a platform, such as Android, lets benchmarking applications run at full potential, with the purpose of evaluating the system's processing power.

Processing power is managed by the operating system. That is why, when comparing hardware alone, it's important to remove the operating system's influence.

Comparing different platforms in an objective, fair way is very difficult, if possible at all.



Android may choose, depending on background applications running for example, to allow an application to work in a certain environment. There are too many variables to consider, but the processing power is used by (1) the operating system, which distributes it to (2) applications.

Two devices with the same processing power, but with different platforms, will perform differently from an application's point of view, depending on each OS's optimization - background processes, save battery settings, the platform's way to handle single windows, etc.

Samsung uses their hardware in a different way than Apple's, and this doesn't make Apple's way a standard or anything.

First I am in no way stating that Apple's way is a standard. not even using them as an example. But the results are for the processor alone, would they have a phone model attached to the result?

And how can one explain the two greatly different benchmark scores, using the same exact benchmark tests with a different package name (as Geekbench did to uncover the "cheating")?

Here's the article BTW:

http://arstechnica.com/gadgets/2013...rking-adjustments-inflate-scores-by-up-to-20/

----------

Overclocking causes the processor to operate at faster frequencies. Samsung is not overclocking the processor of the Note 3 in any benchmark.
Allowing the stock processor to run at its frequency, disabling some Android's management limitations for the purpose of comparing the hardware alone, does not imply overclocking. Not sure what you're asking there.

no, they are not overclocking the processor... but they are forcing the processor to run at a speed that it does not run at for any other application (according to the article). They were doing this when the Geekbench (or a list of other known benchmarking apps) was launched. They discovered this by taking the existing geekbench 3 app, repackaging it with a new name... the SAME EXACT APP... and got results ~20% lower. That does sound like shenanigans to me.

 

[BaldiMac]

Overclocking causes the processor to operate at faster frequencies. Samsung is not overclocking the processor of the Note 3 in any benchmark.
Allowing the stock processor to run at its frequency, disabling some Android's management limitations for the purpose of comparing the hardware alone, does not imply overclocking. Not sure what you're asking there.

"to identify benchmarking apps by name to boost CPU clock speeds". Sounds like they are operating at higher frequencies to me.

 

[Taz Mangus]

First I am in no way stating that Apple's way is a standard. not even using them as an example. But the results are for the processor alone, would they have a phone model attached to the result?

And how can one explain the two greatly different benchmark scores, using the same exact benchmark tests with a different package name (as Geekbench did to uncover the "cheating")?

Here's the article BTW:

http://arstechnica.com/gadgets/2013...rking-adjustments-inflate-scores-by-up-to-20/

All this gaming won't matter anymore as the manufacturers won't know the name of the bench tests as they are going to be stealth named at the time the test is run.

----------

"to identify benchmarking apps by name to boost CPU clock speeds". Sounds like they are operating at higher frequencies to me.

They are operating at higher then nominal freq but not higher then spec'd frequency which is overclocking. In other words, Samsung did not run the CPU higher then 2.3GHz. Still the boost is not the nominal CPU frequency that Samsung runs the OS and majority of the apps at.

 

[samcraig]

You don't even seem to be acknowledging what Samsung is actually doing. Let alone the purpose in comparing smartphone benchmarks. Not sure where to go from here.



As I suspected, you don't know what the difference is, if there is a difference at all.

I am not saying if he is correct or not. But I think his point might be (and I could be wrong) is that whoever creates a benchmark or reading that benchmark is setting some "arbitrary" criteria and applying across the board. Samsung isn't the one doing the comparison - they are showcasing their hardware in the best possible light.

Ultimately this is why I will never understand those so bent on benchmarks btw. Because they have nothing to do with real world use cases. The are a very select group of tests that are supposed to give some "big" picture.

(a bad analogy) - it's like giving a few math problems to a bunch of kids and ranking their math skills on just those questions. Some might be fast and accurate. Some might be slow and accurate. Some might excel at other aspects of math not being test but would be labeled "slow."

Meh... I don't condone Samsung's actions. But I also don't condemn them either (I think). It's all in the interpretation of what that benchmark is going after. Is it real world use case or is it a test of what the hardware and software can do?

 

[BaldiMac]

They are operating at higher then nominal freq but not higher then spec'd frequency which is overclocking. In other words, Samsung did not run the CPU higher then 2.3GHz. Still the boost is not the nominal CPU frequency that Samsung runs the OS and majority of the apps at.

So, overclocking is based on the spec'd frequency of the chip and not the nominal frequency that the device manufacturer chooses to run it at. That makes sense.

Out of curiosity, how would the terminology apply to an A-series processor from Apple. Obviously, it runs at different speeds on the iPhone and iPad.

 

[Taz Mangus]

So, overclocking is based on the spec'd frequency of the chip and not the nominal frequency that the device manufacturer chooses to run it at. That makes sense.

Out of curiosity, how would the terminology apply to an A-series processor from Apple. Obviously, it runs at different speeds on the iPhone and iPad.

I have not seen the spec'd frequency of the A6 or A7 processors. I have read that the nominal freq of the A7 CPU in the iPhone 5S is 1.29GHz.

http://pdadb.net/index.php?m=cpu&id=a70000x&c=apple_a7_apl0698

According to that website the spec'd freq for the A7 is 1.5GHz. That means Apple is running the A7 in the iPhone 5S at 14% lower then spec'd.

 

[BaldiMac]

I am not saying if he is correct or not. But I think his point might be (and I could be wrong) is that whoever creates a benchmark or reading that benchmark is setting some "arbitrary" criteria and applying across the board. Samsung isn't the one doing the comparison - they are showcasing their hardware in the best possible light.

Sure. I understand his point, I just don't think it applies to the specifics of the situation. Smartphone benchmarks, such as Geekbench, have been traditionally used to compare the CPU performance operating at "nominal frequencies" (as defined by Taz Mangus a couple posts ago). Samsung has slipped in extra code to boost performance of the CPU when running benchmark apps to the full spec'd frequency. This grossly exaggerates the performance of the Note 3 relative to other phones. No reason to do this other than to deceive.