Phones Samsung tweaking benchmark scores with an overclock when running benchmark apps?

Discussion in 'Alternatives to iOS and iOS Devices' started by ChazUK, Jul 30, 2013.

  1. ChazUK, Jul 30, 2013
    Last edited: Jul 30, 2013

    ChazUK macrumors 603

    ChazUK

    Joined:
    Feb 3, 2008
    Location:
    Essex (UK)
    #1
    Original Source: Definitely worth a full read!
    ----

    Synthetic benchmarks are never a good indication of real world performance on any handset IMO but it appears Samsung is being shady when it comes to Android benchmark applications.

    http://www.slashgear.com/samsung-ga...tuned-for-benchmarks-research-spots-30292185/

    A very sneaky move from them IMO.

    Benchmarks are okay for some but the way I've seen them gamed over the years by ATI and Nvidia over the years on PC (Quack3 Anyone?) makes me think of them as a more secondary metric over other aspects (design, features, price).

    Peoples' thoughts on this? Are Samsung being underhanded or is there a legit reason for higher clockspeeds when running a benchmark app?
     
  2. ChazUK thread starter macrumors 603

    ChazUK

    Joined:
    Feb 3, 2008
    Location:
    Essex (UK)
    #3
  3. Assault macrumors 6502a

    Assault

    Joined:
    Mar 19, 2013
    Location:
    in the taint
    #4
    Wow. Like this really matters? Did not seem to matter to anyone before when Apple "lied" about the iPhone 5's CPU being twice as fast as the A5X in the 4S? Actual testing revealed, the iPhone 5 was neither twice as fast, nor faster than the 4S at all in many CPU/GPU aspects. In fact, the 5 was slower in things like speech recognition and display rendering.

    I could care less if anyone writes code specific to certain benchmarks. Real world testing is where the proof is. Not in Anadtech's lab. This story is about as stupid as things get, which is probably why it is all over AppleInsider.
     
  4. MRU macrumors demi-god

    MRU

    Joined:
    Aug 23, 2005
    Location:
    Ireland
    #5
    Basically further proof that synthetic benchmarks and end user experience are often a million miles apart...
     
  5. BaldiMac macrumors 604

    BaldiMac

    Joined:
    Jan 24, 2008
    #6
    :confused: Yes, there is a big difference between cheating a benchmark and your made up allegation of a "lie" by Apple.

    Yeah! Who cares if you cheat on benchmarks?! There are other ways to compare! **Unless they cheat on those too**

    It's not like we shouldn't trust them just because they are being dishonest.
     
  6. lazard macrumors 65816

    Joined:
    Jul 23, 2012
    #7
    big deal. People tweak settings/overclock all the time to get optimized benchmark scores.
     
  7. ChazUK thread starter macrumors 603

    ChazUK

    Joined:
    Feb 3, 2008
    Location:
    Essex (UK)
    #8
    Some people take synthetic benchmarks seriously (hence the necessity for Samsung to game the system like they have).

    If they weren't at all important, I doubt that "benchmark booster" code would even exist.

    I've seen the same thing over the years from Nvidia, ATi, 3dfx and others so this is nothing new (but most of that was driver software tweaks rather than hardware overclocks).

    It is funny to see the smartphone works of benchmarking emulating what PC component manufactures were doing years ago.
     
  8. MRU macrumors demi-god

    MRU

    Joined:
    Aug 23, 2005
    Location:
    Ireland
    #9
    Yep sadly this is very true...

    How many youtube videos, blogs, hands on all judge a devices 'worthiness' via AnTuTu and Quadrant scores? Answer = a lot...... And despite evidence that these synthetic benchmarks aren't really the best way to judge end user experience.

    they still use them, even on major review sites....

    We so need some way of benchmarking, but these old legacy apps are probably not the best way and we need someone to do a complete rethink.
     
  9. ChazUK thread starter macrumors 603

    ChazUK

    Joined:
    Feb 3, 2008
    Location:
    Essex (UK)
    #10
    Definitely! :D

    Sadly, lots of people like it back things up with arbitrary numbers that sum up to very little and benchmark scores are a good number to band about when they're on your side.

    There have been plenty of heated discussions on these very boards over GLBenchmark scores and GPU's so I'm not too sure why people would be so dismissive over Samsung getting caught out like this. :(

    Like I said, the very fact that Sammy went to the effort of doing this alone is quite telling IMO.
     
  10. Assault macrumors 6502a

    Assault

    Joined:
    Mar 19, 2013
    Location:
    in the taint
    #11
    Really?
     

    Attached Files:

  11. Sensamic macrumors 68030

    Joined:
    Mar 26, 2010
    #12
    I always disliked Samsung, even though I have an SGS3.

    I've only bought two products from them and don't plan on buying more in the future.
     
  12. lazard macrumors 65816

    Joined:
    Jul 23, 2012
    #13
    It's not only component manufacturers who do it. Just go look at the scores on benchmarks like FutureMark. Almost all the benchmark runs are done by regular Joes with overclocked CPUs/GPUs/RAM.
     
  13. Dr McKay macrumors 68040

    Dr McKay

    Joined:
    Aug 11, 2010
    Location:
    Kirkland
    #14
    Plenty of people cheat benchmarks. People have been known to put a phone in the freezer prior to running the benchmark etc, this is another level of cheating. It's why I don't pay much attention to benchmarking.
     
  14. Assault macrumors 6502a

    Assault

    Joined:
    Mar 19, 2013
    Location:
    in the taint
    #15
    It doesn't matter because the S4 and all follow on Android flagship phones are, and will be, noticeably faster than any iphone (just the nature of Android, not a slam on Apple). There is no comparison being done here.
    Not that benchmarks were accurate to begin with, and you certainly can't compare iOS and Android via benchmarks, because it is impossible to do so with any accuracy.

    So why do we care again? Or is this just some way for the anti-Samsung crowd to try and throw crap at the internet wall? The only thing that should matter is how fast the phone is to you! That is the real benchmark.
     
  15. ChazUK thread starter macrumors 603

    ChazUK

    Joined:
    Feb 3, 2008
    Location:
    Essex (UK)
    #16
    Yes but I do feel it's different a single user tweaking their system for a higher 3DMark score versus the manufacturer actively lowering visual quality in drivers whenever they see a benchmark application is run (or in this case, to use higher than their maximum use level clock speeds for a higher benchmark score).

    The guys and girls at Futuremark got wise to this which is why they have the approved drivers programme.

    http://www.futuremark.com/support/approved-drivers

    The whole community element around Futuremark is pretty much about tweaking and maximising the performance out of a users PC.
     
  16. MRU, Jul 30, 2013
    Last edited: Jul 30, 2013

    MRU macrumors demi-god

    MRU

    Joined:
    Aug 23, 2005
    Location:
    Ireland
    #17
    But....


    1 input = if we don't care, but clearly Samsung do care enough when they

    2 inputed code to enhance synthetic benchmark.

    and if

    3 = its an Anti-Samsung conspiracy and if real world benchmarks are the only thing that is important then

    4 = return to 1

    ;) ad-nauseum



    We do need some form of benchmarking. benchmarking itself gives us some way of measuring improvement of components, but clearly the current format is flawed and perhaps needs a rethink. :)

    Oh and I wouldn't be surprised if all manufacturers (not just Samsung) are doing something similar to maximise AnTuTu and Quadrant scores somewhat artificially (or at least not fully representative).
     
  17. BaldiMac macrumors 604

    BaldiMac

    Joined:
    Jan 24, 2008
    #18
    Really! Are you going to pretend the words "up to" aren't there?
     
  18. ChazUK thread starter macrumors 603

    ChazUK

    Joined:
    Feb 3, 2008
    Location:
    Essex (UK)
    #19
    At what point did anyone bring the iPhone into this? This was never about Apple, HTC, Microsoft or anyone else in particular,

    You may have been making the association because of some shoddy writing by someone at AppleInsider, I don't know, I don't read the crud they put out.

    The big thing is that any Android OEM using the same SOC could be falsely perceived as putting out inferior hardware if two devices are benched with the same advertised base clock speeds which is what I personally think Samsung are up to here.

    I'm not anti-Samsung at all either and have spent a good few pounds on their hardware (Samsung SSD 840, Galaxy S 1,2 & 3, Galaxy Nexus, Samsung Chrome book, Nexus S).

    I definitely agree that real world usage is the only benchmark that should resonate with people but synthetic benchmarks do influence people and Samsung must know this.
     
  19. BaldiMac macrumors 604

    BaldiMac

    Joined:
    Jan 24, 2008
    #20
    :confused: Ahh... The "other people do it" excuse. They cheated. Obviously it's a big deal to Samsung, or they wouldn't have put in the extra work to do it.
     
  20. ChazUK thread starter macrumors 603

    ChazUK

    Joined:
    Feb 3, 2008
    Location:
    Essex (UK)
    #21
  21. Stuntman06 macrumors 6502a

    Stuntman06

    Joined:
    Sep 19, 2011
    Location:
    Metro Vancouver, B.C, Canada
    #22
    Samsung's biggest mistake here is that they got caught. "Benchmark Booster"? Really? They deserved to get caught. It's like planning to rob a bank and your car has the words, "Getaway Car" plastered all over.

    No matter what benchmark system you use, people and companies will be able to find a way to artificially inflate these numbers. It's been done for PC's in the past, so I am not surprised it is happening now on mobile devices. These scores are important, so boosting them is important for companies. It's wrong to game the system, but I don't expect any company to not try doing it.
     
  22. lazard macrumors 65816

    Joined:
    Jul 23, 2012
    #23
    if you ain't cheatin you ain't tryin
     
  23. Fernandez21 macrumors 68040

    Fernandez21

    Joined:
    Jun 16, 2010
    #24
    That seems to be the MO anytime anyone brings up Samsung in a negative light. People either start accusing Apple or apple fanboys or start bashing HTC, doesnt matter if either company was brought up originally. I dont know why people seem to take these things so personally.
     
  24. Shuri macrumors 6502

    Joined:
    Nov 23, 2011
    #25
    Well that is pretty easy to explain.

    Human beings (and monkeys) have very strong evolved Brains for two reasons; first one is scamming others and second one preventing from being scammed.

    It's the second part mixing up with the misbelieve, that there has to be the ONE AND ONLY answer, proven by irrefutable facts, for every question.

    So one of these Questions might be, which one is the best Smartphone maker around. And by stating, that there is only one possible answer, being true for everybody, we get two groups of people; those who buy those magical phones and those who don't.

    Now because people don't want to feel betrayed, they must prove to themselves that they are in the first group of people, since they spend a lot of money, which requires to denounce the others and put them into the other group.

    This is one of the things proving how stupid people really are... :rolleyes:
    And I don't tend to exclude myself of this stupidity.
     

Share This Page