Consumer Reports Retesting MacBook Pro Battery Life After Apple Says Safari Bug to Blame

[bryan.cfii]

The classic "Apple is evil" MacRumors post. They performed a test in a way users don't. They turned browser caching off, which will obviously reduce battery life, and guess what, it reduced battery life. Old joke "Doctor, doctor, it hurts when I poke a spoon in my eye". Don't poke a spoon in your eye then.
[doublepost=1484078711][/doublepost]
It's a hidden Safari setting that you only see if you turn the "Develop" menu on. Normal users will never see this setting. For good reason.

Doesn't Private mode use this feature too? Or am I wrong?

 

[BorderingOn]

Consumer Reports sucks. The fact that they say they turn off caching to work the batteries harder, but are also trying to show consumers what they can expect is a classic red herring. I would bet significant money that the typical Consumer Reports reader would NEVER think to turn off page caching in Safari, let alone actually know how to do it, so CR doing so was setting up an unrealistic / uncommon scenario for what people could expect.

Apple should sue them, to recover the $18,000.00 in lost revenues caused by the 10 CR readers who bought a PC instead of a new rMBP.

Really? Do you sit for hours refreshing the same content or do you browse new content? It was a simulation of browsing for a battery test. How would you have done it?

If the bug wasn't present, the test would have run fine.

 

[carrrrrlos]

Because developers are not pro consumers.

 

[BorderingOn]

I understand the desire by CR, but that isn't actual usage. If they wanted to mimic how most of us use the machine they'd not turn off the caching. While its good that apple fixed the bug, it seems kind of sketchy for CR to do that

Then how would they have ensured any consistency in the results? They clearly stated how the test was performed.

 

[wallysb01]

So, Apple was right. That is the bottom line. Man, this place has become negative beyond belief.

Not really. CR disabled caching in order to simulate reloading pages, rather than just pulling from the cache, and there was some sort of bug in Safari if you do that. At least that's what Apple is saying. If that's correct, its still Apple's fault. A feature of their software didn't work. This wasn't CR's fault, and apparently this has been CR's method for testing battery life for a while now...

 

[Juicy Box]

Sorry, I don't really feel like losing 500 GB of data again just because the Guest Account is on.

Overall 10.6 was great, it is not like every version of Snow Leopard wiped data, plus it was only while using guest account, so many people didn't even know about the problem. Hopefully you had a backup.

10.6 was much more stable than the recent versions. Out of all the OSX version since Snow Leopard, I think Mountain Lion was the best, but all the other ones were mediocre in comparison. Still better than Window 7 though.

 

[blackberrycubed]

Apple must have put a lot of pressure to make CR crumble like this....

 

[triton100]

And others who have tested their laptop under the conditions Apple specified, have.

I ran tests under all of the conditions apple specified as well as my own. How could I quantify my findings without having done that

 

[macfacts]

I think theyre both culpable. Apple for the bug, but CR for running a non-real world sim to test battery life and publish as real world.

In the test, they want the test to be consistent, so it is fair and so they can compare between brands. That's why they used the same 10 pages on their test web server.

But normal people don't visit the same pages, so they disabled caching to make the test more realistic.

Now do you understand why they used that setting?

 

[Johns12]

Why are people so lazy and dense? Why can't people read and comprehend the written word? Let me spell it out really clearly.

Apple makes a claim that you get X hours browsing on the MBP.

Consumer Reports wants to validate that claim.

Consumer Reports devises a strategy to simulate browsing.
Issue #1: different web sites download at different speeds
Issue #2: I can't have a person sit there and randomly visit websites because who is to say that they will get a random sampling of websites that are fast and slow, simple or complex, etc.
Solution: Simply create a website that has average speed, and average complexity and load it.
Problem. Browsers cache content so after the initial load, refreshing doesn't use much power.
Solution. Turn of caching. Now refreshing a web page of average speed and complexity over and over simulates randomly browsing random sites

TL/DR Consumer reports methodology was just fine, I defy you to come up with an alternative method for making a measure of average power consumption visiting average websites until the computer runs out of power. They did acknowledge that the results are somewhat harder than real life, because obviously some sites we go to are cached. But the results are the same for every machine they test.

What then happened is that using this very logical testing process, Consumer Reports uncovered a bug with how Safari handles icons. This resulted in the same test in Safari using up much more power that the test in Chrome.

If Consumer Reports redoes the test using Safari, they will NOT change the methodology, the claim is that Apple fixed the bug in Safari and now with the cache disabled, the test will work as expected. Nobody is asking CR to change the test for Apple, only to do it on the fixed software.

The only "fault" here lies in the fact that Apple had a bug that was exposed by Consumer Reports very logical testing procedure. AND that Apple did not want to comment until they figured out what was causing the problem.

I don't think Consumer Reports has a responsibility to wait for Apple before posting their results. If anything, they did more than they needed to by letting Apple know what they found.

Who knows what other bugs are waiting for just the right set of circumstances to wreak havoc.

Now I will sit back an watch post after post come that ignores these fact in the name of apple hate or apple love.

Thank you. My head was about to explode with some of the crazy comments here. When ANY product is testing, real world usage is not the measure. It is a synthetic test. It has to be for repeatability.

 

[triton100]

In the 15" version, the previous MBP had a battery that was 31% larger. Everything else equal, that same pro music application would have lasted a bit less than 4 hours, in a computer rated for 9 hours. I don't remember people walking around with pitchforks back then.

The real problem is not the exact number but that things got worse in quite a number of use cases in a noticeable way. They also got better in some use cases but even if the average battery life over all users stayed the same, worse for some, better for others, the voices of those at the loosing end will always be much louder than of those at the winning end.

Yea this is all true. I was just very disappointed at only having 3 hours. The 2015 you're right gave about 4. But even that one hour makes a difference between being tethered to a power cable or free to roam as inspiration takes you. Hence the lack of pitch forks surrounding the earlier model.

 

[sudo1996]

Why are they "fair now"?

And why were they "garbage" back then?

They are Consumer Reports, not Apple Public Relations or Apple Marketing. It is not their job- and presumably not yours- to help sell Apple products. How can they be wrong when they have something negative to say but "fair" or right when they have something positive to say? It's likely the same people, doing the same job, probably running the same tests, etc.

Well, in both cases, they had something negative about Apple to say. The first time, they were recommending against buying the best smartphones on the market (iPhone 1, 3G, 3GS, and 4) due to some weird use case antenna problem. The second time, they were recommending against buying a glitchy and nerfed MacBook Pro due to a battery problem that should affect most users... except that they (reasonably) assumed Apple's "disable caching" feature worked properly.

 

[moonjelly]

The problem was because of a bug, not because of the use of a hidden setting or being used wrong.

If CR was wrong in using that setting, why is Apple releasing a fix?

Just the setting did not trigger the bug. Setting + canned test triggered a bug. Apple did not say it was wrong to turn off caching. They did not blame CR. They said it was obscure and would not affect users. And they fixed it. But that is all too professional and rational to fill the need for drama on internet forums.

 

[sudo1996]

Not really. CR disabled caching in order to simulate reloading pages, rather than just pulling from the cache, and there was some sort of bug in Safari if you do that. At least that's what Apple is saying. If that's correct, its still Apple's fault. A feature of their software didn't work. This wasn't CR's fault, and apparently this has been CR's method for testing battery life for a while now...

Apple's only fault is that the cache disable feature doesn't work right. Normal users don't disable the cache, only testers. Apple screwed over CR, not the consumers. It's fair for CR to retest with the minor bug fixed.

Reminds me of times in college when I submitted project code but gave some function the wrong name and got to fix it instead of taking a 0...

 

[MH01]

Some people are such hypocrites. When CR runs the same tests in the past, and have given Apple laptop high praise, people agree with the testing, when CR gives one laptop a bad rating, suddenly they are crap. No the issue is not with CR but complete lack of objectivity of some members, same old legion of apologists working overtime.

 

[wallysb01]

Apple's only fault is that the cache disable feature doesn't work right. Normal users don't disable the cache, only testers. Apple screwed over CR, not the consumers.

Having a bug screwed Apple by handing themselves some negative press. CR really had nothing to do with it at all, really; they were just the messenger.

 

[moonjelly]

Total clarity. Absolutely spot on.

[In my opinion, CU uses credible engineering behind all their tests to provide unbiased comparison among brands. Where they may lapse is on their reliability ratings, which are based on statistical synthesis of responses from subscribers to annual questionnaires -- CU does not do long-term product testing, because of obvious economic constraints, and that may be their only fail.]

Agree, except they do have bias. Even if they do not have a brand bias, they still have bias.
A contrived car example:
CR rates sports cars in 5 categories: initial cost, initial quality, economy, reliability, performance.
Sports car weekly rates sports cars: acceleration, slalom, top speed, braking, overall value.

See how both are biased?

 

[tech4all]

You must be a PRO user, huh?

Don't you love the "you don't want a Mac cause you can't afford it" nonsense? Haha

 

[GoodWheaties]

"Are you expecting CR to look thru source code they don't have? Do some software profiling? Run a debugger?"

Or even wonder why CR's tests which are properly being executed (and presumably monitored, observed, and supervised) were able to pull fresh web pages off their server, in some trials, for 12, 14, 16, 18 1/2, and 19 1/2 hours straight?

Why would that be so strange? I just got over 18 hours of usage (and over 4 days standby) from a nearly 2 year old iPad Air 2 using mainly Safari and Youtube.

 

[mw360]

Apple isn't hinting. Apple is stating as forcefully as possible (while remaining polite) that the test settings were odd and obscure - not "rigged", because there was no intent to get bad test results. It makes sense to try to simulate a power user - but simulating a power user is not the same as being a power user. The bad test results happened only for a _simulated_ power user. They don't happen for a real one.

They aren't simulating a power user. They're turning off caching, not because users do that, but because loading all their test pages from the browser cache wouldn't be a test of internet browsing at all - it would be a test of SSD reads.

I'd imagine they aren't trying to simulate a user. Users are erratic and non-repeating. They are trying to approximate a user, in a repeatable way to be fair to every model they test. Every laptop gets the same test, so every laptop loads the same pages. They cycle through the same set of pages so that the power drain is linear, not biased towards particularly strenuous pages at the start or end of the test.

I've seen you post a lot in here. Apologies if I missed it, but what would your protocol be for a test be?

I have some sympathy for Apple, in that they got really unlucky when they were already on the back foot, but the test protocol seems perfectly fine to me, and CR did state it was odd, and almost certainly a software problem with Safari. I don't see why they should have torn their protocol apart when the problem absolutely was an Apple bug which CR were under no obligation to diagnose.

 

[Wild-Bill]

They can do all the battery re-tests they want. At the end of the day, the MacBook 'Pros' are still overpriced and spectacularly underwhelming.

Apple is just having a hard time accepting the fact that they blew it.

 

[nofear1az]

I agree with you, it's fine to alter the cache settings. I'm referring to the fact that CR quickly realized a likely issue with Safari (or at least a very curious anomaly), but rather than hold off on an official opinion until they had more information, they appeared to rush the article as if to meet a deadline. As a result, the article has a laughably simplistic conclusion which almost seems to ignore the excellent battery performance under Chrome.

It would have made more sense to write, "The MacBook Pro achieved an excellent 19 hours battery life using Chrome, although we noticed inconsistent results using Safari, likely due to a software issue. As a result, we recommend monitoring your usage of Safari. If you aren't satisfied with results, switch to Chrome until a fix is issued."

If it were worded like that do you honestly think Apple would have looked deep into the problem to find that bug? Remember, Apple has been denying battery issues with the MacBook ever since it's release no matter what people were saying and it took an embarrassing hit from CR for Apple to look into it more seriously and find that bug; and that nobody knew existed before. I don't mind the way it was panned out considering how quickly it got the root of the problem.

 

[citysnaps]

Why would that be so strange? I just got over 18 hours of usage (and over 4 days standby) from a nearly 2 year old iPad Air 2 using mainly Safari and Youtube.

OK, got it...

Getting 18 hours of continuous real-life usage on a MBP would not be surprising to you, even though Apple only specifies up to 10 hours.

Did Apple blow it with a much too conservative battery life spec?

 

[Billykonstantine]

I got my 13-inch MBP Touch Bar in November. Using it to be online (wifi) reading, watching some videos, I was getting 4-5 hours. That's not acceptable. I returned it on Jan 3, no questions asked...I don't think they wanted customers seeing returns.

Wow that is insane. No way is this acceptable for a pro machine.

 

[DuckyDelite]

"This is not a setting used by customers and does not reflect real-world usage."

********. It's a MacBook Pro. I always have this setting on. Why does Apple think people buy Pro laptops?