Samsung Again Caught Inflating Benchmarking Scores, Phil Schiller Calls 'Shenanigans'

[Burger Thing]

I have just read the full review of Ars Technica. So, the Gallery App takes anywhere from 15s to more than 2 min to open. On a brand new so called top tier mobile device in 2013

Also other issues, which sounds just like my increasingly sluggish Note 1 still seem to apply as well. Quote from Ars Technica:

In terms of performance, the Note 3 is not what I would call "blazing fast." We've run into all sorts of stutters in animation and even outright pauses while testing it. While the hardware is top of the line, Samsung appears to have pushed the software envelope a little too far this time, and it doesn't feel as responsive as other modern Android phones. We've even run into weird error messages like "low memory" warnings when opening GFXBench, despite the 3GB of ram and Android's automatic memory management. While the phone itself sometimes pauses, Samsung isn't helping itself in the "perceived responsiveness" department either, thanks to the way the home button is implemented. You can press it twice to open up S-Voice, which means when you press it once to go Home, it waits for half a beat to make sure you aren't going to press it again to load S-Voice. When the phone temporarily locks up and you frustratedly hit the home button hoping it will wake up and load, every additional home screen press launches some kind of app or feature that isn't the home screen. This probably makes things worse. I've accidentally opened Flipboard (by pressing the home button while on the home screen) so many times I've lost count.

So, we have some serious usability issues, but the software is optimized to show abnormally high benchmarks.

Good to see, that Samsung's software department got the priorities right. LMAO.

Hilarious.

 

[BvizioN]

are you including devices made by other companies that include Samsung components?

I think the guy clearly said "Samsung Devices"
Devices made by other companies that include components designed by other companies and manufactured by Samsung are clearly not Samsung devices.

----------

No one is being a jerk, you stated Samsung products

Again that was SAMSUNG DEVICES.
Learn to pay attention to the details

 

[captain kaos]

I have just read the full review of Ars Technica. So, the Gallery App takes anywhere from 15s to more than 2 min to open. On a brand new so called top tier mobile device in 2013

Also other issues, which sounds just like my increasingly sluggish Note 1 still seem to apply as well. Quote from Ars Technica:



So, we have some serious usability issues, but the software is optimized to show abnormally high benchmarks.

Good to see, that Samsung's software department got the priorities right. LMAO.

Hilarious.

I do remember when i had a SGS3 the stock gallery app took an ice age to load up photos, and that was as soon as i bought it so hardly any photos on it. It was only when i installed a 3rd part gallery app it was ok, but really, for a phone with that sort of power it was pathetic!

 

[damitssam]

Sony z1 > than this ugly piece of junk.

Sony's software got it right, minimalist and simple.

 

[apollo1444]

Yup. On one hand, we have all the fandroids screaming about how quad-core CPUs makes their phones faster and more awesome as if it actually means something. On the other, we now have years of hearing iFans yak their heads off about 64-bit processors, and how it makes Facebook run twice as fast to look forward to

WOOHOO! BOTH SIDES ARE ARMED, AND EVERYONE WITH AN OUNCE OF COMMON SENSE LOSES! WELCOME TO THE MOBILE BATTLEFIELD, EVERYONE!

except for the fact that the A7 is a dual core processor that rapes the snapdragon 800 the Note 3 has.

 

[MacinDoc]

You know for a fact the device in question never pushes more voltage to throttle up when the load increases? Please provide your in depth tests of cpu clock speeds and voltages when engaging the phone in demanding tasks.

We know for a fact that the upclocking is triggered by the detection of certain strings indicating that a specified benchmark (or an app renamed with the name of one of those benchmarks) is running. If it were triggered by load detection, then the benchmark name detection strings would be unnecessary. This is consistent with the detailed testing done by Ars that found that core apps were not accelerated (and some were downright sluggish according to Ars), and benchmarks, when renamed, were not accelerated.

So, I guess if you rename whatever app you want to run faster to "Geekbench 3", then you can enjoy the increased performance. Keep in mind, though, that there is probably a reason why Samsung does not run the rest of the apps at this speed, likely because it will either make the device less stable, or much more rapidly drain the battery. Of course, if you have a number of apps you want to run faster, then you may quickly run out of benchmark tests to name them after, and it might be hard to keep track of which is which. I guess you could always root your device and have the accelerated mode always on, but good luck with your battery life.

On the other hand, I suspect that the majority of iPhone users prefer a device that just works the way it is supposed to and performs well right out of the box, without the hassle of rooting it, and that doesn't have to be plugged in to use it...

 

[Renzatic]

except for the fact that the A7 is a dual core processor that rapes the snapdragon 800 the Note 3 has.

It doesn't rape it, but it is faster. That's not directly due to the 64-bitness of it though, rather the fact it's faster processor overall.

 

[RonDMC]

With Apple greed knows no limits.

So because they are successful and have lots of coin to show for that success they are suddenly greedy and should give it all away? Hope you're never in charge of a company, won't be in business very long!

 

[LostSoul80]

Who cares? It's the result of benchmark produced by an application.
I don't see what's the deal.
I don't buy a phone because it has good benchmarks.

 

[phoenixsan]

I dont get......

Why tamper if you can build a group to do that for you:

http://www.engadget.com/2013/09/26/mobilebench-benchmarks-samsung/

Really, really, a moronic undertaking.

the remark....No need to offense tough, if the "moronic" undertaking is coming from me. In that case, you can go to hell......

 

[CFreymarc]

And no other company has ever done this?

Actually this is a trick from the graphic cards days back in the 90's before the PCI standards took hold. A few graphics card manufactures had their drivers look for the benchmark threads running in the OS and returned false results for the tests. Took a few process servers to the homes of of the "t-shirt at the keyboard" and some Federal trade fraud allegations for it to stop.

----------

I do remember when i had a SGS3 the stock gallery app took an ice age to load up photos, and that was as soon as i bought it so hardly any photos on it. It was only when i installed a 3rd part gallery app it was ok, but really, for a phone with that sort of power it was pathetic!

Goes to show you it is not all hardware but the talent you hire for the code. I've seen a few projects wrecked with semi- or non-technical management hiring cheaper, sub-standard and often outsourced "talent" to handle a job the agent was lying through their teeth about to close the deal.

One contract a few years ago, saw the train wreck coming two towns away with the "A crew" selling and the "C crew" implementing. After one of those talking-to-the-wall manager discussions, myself and another senior engineer resigned over the project's execution.

Six months later, the code delivered wasn't even one tenth of the performance specified and the project was canceled with a lot of money wasted. To top it off, one personnel manager called trying to have me admit some sort of fault to have legal action commence to recover loss. It ended up going no where and we both moved on to better run outfits.

 

[mylop]

A lot of effort/risk for Samsung to go to to artificially increase a figure that 99% of potential customers don't care about, don't understand and probably will never hear about anyway.

 

[jasvncnt]

On the other hand, I suspect that the majority of iPhone users prefer a device that just works the way it is supposed to and performs well right out of the box, without the hassle of rooting it, and that doesn't have to be plugged in to use it...

Whats the majority? Most I know Jailbreak to get more out of their device.

 

[samcraig]

Whats the majority? Most I know Jailbreak to get more out of their device.

Is he implying that the majority of Android users need to root their phones to have functionality they want? He couldn't possibly...

And that the phone needs to be constantly plugged in to work? He couldn't possibly...

 

[tbrinkma]

Does this seem more of a feature only to me?
When a particularly demanding task, like a benchmark, is executed, the CPU releases more power. I call this "efficiency".
And these numbers came out from the CPU's calculating power, not from nowhere.. how is this fake?

It's not "when a particularly demanding task is executed". It's "when this benchmark is executed". Since a benchmark is intended to gauge the performance of the system, if the benchmark gets 'special treatment', then the benchmark isn't providing useful results. To do this, and then tout the results, indicates a special level of dishonesty.

This is equivalent to 'hypermiling' for a car's EPA mileage test, and then advertising using *those* results instead of the much lower results you get when you actually follow the EPA test standards.

 

[magbarn]

Since when was running your 2.3GHz CPU at 2.3GHz cpu to test the limits cheating? they did NOT overclock the CPU at any point during the test.
........
similarly im not going to drive my Model S at the limit to get the full 443lbft of torque at every single stop light, but for a 0-60 or dyno test, sure, ill do it once or twice.

you guys need to relax, apple already got caught "cheating" when it comes to ipad benchmarks.

almost all Note3's are capable of running at 2.3GHz for a short time and actually achieve the benchmark scores, apple on the other hand made up the 4x faster than the tegra 3 score and third parties cannot verify their claim.

most android users that are knowledgeable would have better scores than what Samsung puts out anyways since you can root the note and overclock it to whatever you want (within reason)

So you wouldn't be a tad bit disappointed if Elon Musk sent out model S' with an 'unlocked' 443 lb ft mode for magazine testing to spank the M5 and then sold you a car that really was locked at 300 lb ft for battery longevity issues?

According to the posted article, this super speed mode only ever came on for benchmarks and no other apps activated this mode.

 

[duffman9000]

We know for a fact that the upclocking is triggered by the detection of certain strings indicating that a specified benchmark (or an app renamed with the name of one of those benchmarks) is running. If it were triggered by load detection, then the benchmark name detection strings would be unnecessary. This is consistent with the detailed testing done by Ars that found that core apps were not accelerated (and some were downright sluggish according to Ars), and benchmarks, when renamed, were not accelerated...

IIRC, AnandTech talked about this during a podcast when they analyzed the SGS4. Basically, mobile should copy what Intel did with its Core turbo modes. Doing it in software is a bad idea.

We don't know if Apple is already doing this*. Probably.

*Implemented something similar to Intel's Turbo feature.

 

[Lone Deranger]

Who cares? It's the result of benchmark produced by an application.
I don't see what's the deal.
I don't buy a phone because it has good benchmarks.

Many people don't pay attention to benchmarks. But review websites and sales personnel often do. And when these sources start selling the product with sales pitches as "fastest phone on the market" and "faster than iPhone5S", etc. because this benchmark said so, a far larger audience is sucked into the deceit. Samsung knows this and that's what they're gunning for.

 

[samcraig]

Many people don't pay attention to benchmarks. But review websites and sales personnel often do. And when these sources start selling the product with sales pitches as "fastest phone on the market" and "faster than iPhone5S", etc. because this benchmark said so, a far larger audience is sucked into the deceit. Samsung knows this and that's what they're gunning for.

I think that's of interest to gamers. Other than that - what do people do on their phone that requires something so lightening "fast."

There's a good chunk of people who just call, text, email, use some apps that don't really benefit from a faster phone. Oh yes - overall experience using the device might be a little snappier.

It's like people at the store that get suckered into faster computers when all they do is surf the net. They don't need 4/6/8/12 cores. They don't need GIGs of RAM.

Same thing with Digital Cameras. So many people believe it's all about megapixels - which is funny since a lot of people are using their cameras primarily for photo sharing online. You don't need a 21 megapixel camera.

You also don't need a TV with a contrast ratio of 100,000,000:1

So yes - it's clear to see why Samsung would want to appear faster. Which is a little odd because their phones are already pretty darn fast anyway. It's not like they need the "#" on paper to prove it.

Whenever I'm asked about devices by peers or family - I never suggest by specs. I listen to what they want and then shortlist to a few devices (phones, cameras, tvs, etc) and tell them to go play with devices themselves and see what feels best for them.

That's all that really matters.

 

[TheHateMachine]

Read the Ars article.

Reread my post and the guy I was replying to.

He stated that the phone never throttled its cpu up ever. Even on demanding apps. According to him It just sits at its idle clock and voltage rate all the time and only throttles up in a benchmark. This is simply untrue.

----------

We know for a fact that the upclocking is triggered by the detection of certain strings indicating that a specified benchmark (or an app renamed with the name of one of those benchmarks) is running. If it were triggered by load detection, then the benchmark name detection strings would be unnecessary. This is consistent with the detailed testing done by Ars that found that core apps were not accelerated (and some were downright sluggish according to Ars), and benchmarks, when renamed, were not accelerated.

So, I guess if you rename whatever app you want to run faster to "Geekbench 3", then you can enjoy the increased performance. Keep in mind, though, that there is probably a reason why Samsung does not run the rest of the apps at this speed, likely because it will either make the device less stable, or much more rapidly drain the battery. Of course, if you have a number of apps you want to run faster, then you may quickly run out of benchmark tests to name them after, and it might be hard to keep track of which is which. I guess you could always root your device and have the accelerated mode always on, but good luck with your battery life.

On the other hand, I suspect that the majority of iPhone users prefer a device that just works the way it is supposed to and performs well right out of the box, without the hassle of rooting it, and that doesn't have to be plugged in to use it...

Core apps outside of the camera did not trigger a higher frequency in the CPU. However these phones don't just sit at idle speeds all the time like the guy I was replying to was suggesting. If you load a demanding game for instance the devices start ramping up their processors. Maybe not to the level of this benchmark boost but the guy I was replying to stated the phones never ramp up during load and only during benchmarks. This is untrue.

 

[jayselle]

IIRC, AnandTech talked about this during a podcast when they analyzed the SGS4. Basically, mobile should copy what Intel did with its Core turbo modes. Doing it in software is a bad idea.

We don't know if Apple is already doing this. Probably.

There is zero chance Apple is doing this. They might be out for huge profit margins, but they aren't deceptive. The fact that Phil is calling Shenanigans is proof enough.

Apple has never been about specs. They're about the user experience. Based on the Note 3 review it has a sluggish feel at times. That's what matters. Not synthetic benchmarks to make yourself feel better. I've yet to hear anyone say the iPhone 5S feels sluggish.

 

[giantfan1224]

So you don't care that it is manufactured by Samsung essentially but stamped with an Apple logo? I'd recommend getting rid of the Samsung parts. Otherwise, you're buying Samsung. Clearly, you don't want to support them, or something or other.

I'm not the one that made the original statement. I'm just pointing out the LARGE difference between a component and a final product. Apple has ultimate control over its supply base and must ensure the quality of all components as they comprise the final product.

With that said, I think Samsung makes fine products. I just don't agree with the way they conduct business, which opinion is further enforced by this very article.

 

[tbrinkma]

This is nothing different than any other company, including Apple. Of course Samsung it trying to show the best case scenario, OF COURSE. APPLE DOES THE SAME DANG THING!

Someone please show me just ONE macbook pro that gets 5 hour battery life under normal use, let alone the "Up to 7 Hours" claimed. I have worked on 1000+ Macbooks, and they DO NOT EVER GET THAT BATTERY LIFE. PERIOD!! So Apple apparently tweaked whatever they could to come up with the 7 hour number. Clearly not realistic.

Take off your fruit shades. Let Apple win with better products, rather than with their executive donkeys.

3rd party battery life tests for Apple's products *routinely* get values in the same neighborhood as the official Apple numbers. Often times, the 3rd party tests give *longer* battery life than the official Apple numbers.

That's easily verified with even a cursory search of reviews.

----------

The point is that Apple cheats on their claims also. IE. Show me a macbook pro with 7 hours of real battery life. PLEASE. I can't find one with 5 hours. But they claim 7. how is that different from Samsung tweaking their product to get the best benchmark in order to claim it.

Good grief I love Apple, but I can not stand the following of people who think Apple is the royal family of businesses. They are just a crooked as the rest and it is hypocritical to make the claim like this.

Mine does. Apple actually provides the details about the system configuration, and usage during testing, which is designed to provide a reasonable baseline approximation of typical user activity during the test. If you mimic those settings, and that usage pattern, you'll find that your battery life is pretty danged close to their advertised numbers. (Barring the aging effects on battery life.)

----------

News sources never mentioned antennagate? foxconn? tax dodging? tracking your location? e-book price fixing? None of those were ever on the news? oh, ok.

To be fair, they rarely reported on any of those 'events' *accurately*.

Most of the Foxconn news, for example, was about groups of employees working on non-Apple products, but Apple was the only client company mentioned in most accounts. (e.g.: The employees on the X-Box line who threatened to jump from the roof as part of their negotiations for better pay and conditions.)

----------

Apple executive Phil Schiller -- senior vice president of Worldwide Marketing and the most prolific tweeter amongst Apple's senior staff -- linked to the Ars article in a tweet, saying only "shenanigans".

Soooooo..... the executive Of Apple publically slanders it's biggest supplier of components...

It's not slander if it's true.

 

[TheHateMachine]

There is zero chance Apple is doing this. They might be out for huge profit margins, but they aren't deceptive. The fact that Phil is calling Shenanigans is proof enough.

Apple has never been about specs. They're about the user experience. Based on the Note 3 review it has a sluggish feel at times. That's what matters. Not synthetic benchmarks to make yourself feel better. I've yet to hear anyone say the iPhone 5S feels sluggish.

Phil is going to say anything to talk down the competition and talk his own stuff up. His word isn't the gospel. Apple may or may not be doing the same things behind the scenes... or something similar. Stating you are 100% certain Apple is not and not being an insider at Apple is kinda silly. You don't know.

Also about the specs thing, this is not really true. Look at the last event for the 5s and how they focused a bit more on its performance and specs. Hell even before that it was about resolution and they aren't too shy on throwing their battery endurance around. Apple whores specs out just like anyone else.

 

[tbrinkma]

Both ATI and nVidia were caught cheating; it's has totally ruined their business.

Actually, for the GPU generations where ATI or nVidia got caught cheating on benchmarks, it *did* hurt their business. It would be less of an impact today, since the majority of their business is on chips already built into someone's systems (laptops), but back when it happened, that wasn't the case. It was largely an enthusiast market at the time.