Consumer Reports Retesting MacBook Pro Battery Life After Apple Says Safari Bug to Blame

[gnasher729]

Why are so many people taken in by Apple hinting that the test settings were somehow odd, obscure or rigged? Turning off the browser cache isn't particularly arcane and makes perfect sense to me if your test involves loading the same pages over and over (which it should, to generate fair comparisons between models). Fetching webpages entirely from the cache over and over isn't exactly real world usage either.

Apple isn't hinting. Apple is stating as forcefully as possible (while remaining polite) that the test settings were odd and obscure - not "rigged", because there was no intent to get bad test results. It makes sense to try to simulate a power user - but simulating a power user is not the same as being a power user. The bad test results happened only for a _simulated_ power user. They don't happen for a real one.

 

[manu chao]

I think theyre both culpable. Apple for the bug, but CR for running a non-real world sim to test battery life and publish as real world.

Every reproducible battery test doesn't reflect the real world. Reproducibility requires automation and automation differs from real use in unavoidable ways.

 

[moonjelly]

Yeah, I completely agree. It seems really odd for them to say that they aim to test it under the same type of conditions a consumer would experience, then go out of their way to enable this hidden setting because they randomly want the browser to pull each page refresh from the server. If that isn't a setting most people will ever enable, and thus not how most consumers will use the laptop, it's pretty arbitrary.

Web page content is extremely dynamic and variable. Visiting the same site could load different videos, or no videos seconds apart. To get consistent results browsing tests use a variety of static pages, clear and reload them. It makes sense as a repeatable test.

I understand the people who are saying they tested it the same as every other laptop. But when the results are tied to specific software, not the hardware, I have to agree it was an opportunity for click bait. Who clicks on good news?

They uncovered a relatively minor software bug, didn't recognize it or look any further. Minor, and even major bugs are not rare in any new OS. Their canned tests would miss the overwhelming majority, but also be asymmetrically skewed if they happen to trigger one. Its their job to be more informed than the average consumer. That is the failing. If you read the initial report the same test also scored 18 hours!

 

[gnasher729]

As a web developer too, I definitely disable cache. So this would affect me.

Do you? Why? Other than complaining on MacRumors, what would be your reason?
And do you use a laptop on battery when you do this?

 

[dapork]

Music to my ears:

http://www.macworld.com/article/315...issues-found-in-consumer-reports-testing.html

“[Consumer Reports’] use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life.”

 

[morphineseason]

Thanks for your helpful beta feedback, Consumer Reports!

Honestly pretty sad that Apple didn't catch this in their "extensive testing", especially considering Safari is the default browser that ships with the Mac and is probably one of THE most used apps.

 

[gnasher729]

That's my point. If CR is testing a dishwasher that just flat out doesn't work, are they going to go to the manufacturer and say hey.. I think maybe there's something wrong with this unit... or do they just publish a report saying don't buy this one, it doesn't work.

Well, I bought an expensive battery-driven vacuum cleaner that just flat out didn't work. Then I figured out that I had plugged it in for charging (hours ago), and it just didn't work when plugged into the charger. Unplugged it, worked. My shaver does the same. In that situation, if it "doesn't work" you better check with the manufacturer.

 

[Naimfan]

Music to my ears:

http://www.macworld.com/article/315...issues-found-in-consumer-reports-testing.html

“[Consumer Reports’] use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life.”

That's not what Consumer Reports said in its statement: "Indeed, when we turned the caching function back on as part of the research we did after publishing our initial findings, the three MacBooks we'd originally tested had consistently high battery life results."

 

[kdarling]

Web page content is extremely dynamic and variable. Visiting the same site could load different videos, or no videos seconds apart. To get consistent results browsing tests use a variety of static pages, clear and reload them. It makes sense as a repeatable test.

Exactly, the test made sense.

In fact, Apple does very much the same thing when they do battery tests. (Serving a set number of pre-stored pages over and over again.)

Except apparently Apple doesn't turn off the cache in their tests. Which is interesting.

Does this mean Apple's battery tests are bogus because they're actually hitting the cache all the time?

Or are they using a different method to make sure each page is actually served.

They uncovered a relatively minor software bug, didn't recognize it or look any further.

It's not up to CR to figure out a bug like this.

Heck, it looks like it took Apple over a week to figure it out... and it's their code.

The upshot is everything is sorted out, and that's all that matters. Only on an Apple fan site would this rate more than a dozen posts.

 

[Nunyabinez]

Why are people so lazy and dense? Why can't people read and comprehend the written word? Let me spell it out really clearly.

Apple makes a claim that you get X hours browsing on the MBP.

Consumer Reports wants to validate that claim.

Consumer Reports devises a strategy to simulate browsing.
Issue #1: different web sites download at different speeds
Issue #2: I can't have a person sit there and randomly visit websites because who is to say that they will get a random sampling of websites that are fast and slow, simple or complex, etc.
Solution: Simply create a website that has average speed, and average complexity and load it.
Problem. Browsers cache content so after the initial load, refreshing doesn't use much power.
Solution. Turn of caching. Now refreshing a web page of average speed and complexity over and over simulates randomly browsing random sites

TL/DR Consumer reports methodology was just fine, I defy you to come up with an alternative method for making a measure of average power consumption visiting average websites until the computer runs out of power. They did acknowledge that the results are somewhat harder than real life, because obviously some sites we go to are cached. But the results are the same for every machine they test.

What then happened is that using this very logical testing process, Consumer Reports uncovered a bug with how Safari handles icons. This resulted in the same test in Safari using up much more power that the test in Chrome.

If Consumer Reports redoes the test using Safari, they will NOT change the methodology, the claim is that Apple fixed the bug in Safari and now with the cache disabled, the test will work as expected. Nobody is asking CR to change the test for Apple, only to do it on the fixed software.

The only "fault" here lies in the fact that Apple had a bug that was exposed by Consumer Reports very logical testing procedure. AND that Apple did not want to comment until they figured out what was causing the problem.

I don't think Consumer Reports has a responsibility to wait for Apple before posting their results. If anything, they did more than they needed to by letting Apple know what they found.

Who knows what other bugs are waiting for just the right set of circumstances to wreak havoc.

Now I will sit back an watch post after post come that ignores these fact in the name of apple hate or apple love.

 

[coolfactor]

I can remember getting 12 hours from my 2013 MacBook Air back when I got it. Now it's running a lot more apps and background processes, so doesn't last nearly as long, but for Apple to be "proud" of a Pro machine lasting less than a consumer machine is surprising.

 

[MacBH928]

Any else surprised that Apple still supporting Safari (a web browser that doesn't make them money) while they are slashing products and standards left and right getting rid of headphone jack, neglecting Macs, dropping all standard ports, displays, and their routers?

 

[gnasher729]

Please for the love of Sweet Baby Jeebus stop. If you're not going to stop, at least attempt to present a factual representation of the situation.
Disabling the cache did not cause a loss of battery life. Because if it did, the laptops would have experienced consistently low battery life. The laptops experienced wildly varying battery life, not just low. .

You need to read more carefully. CR said themselves that without caching disabled, they had consistently long battery life. With caching disabled, they didn't. With the original, unfixed software. And disabling the cache is something that 99% of users don't do, and the one percent does it rarely. So they got results that were irrelevant to real life usage.

The problem was because of a bug, not because of the use of a hidden setting or being used wrong.

If CR was wrong in using that setting, why is Apple releasing a fix?

Using a hidden setting uncovered the bug. So it was a bug that didn't affect normal users. And that wouldn't have affected tests with _real_ heavy users, only tests with simulated heavy users.

Your question "why is Apple releasing a fix" is more than stupid. Because there is a bug, and Apple has found the bug, so it gets fixed. What do you expect them to do? Leave a known bug in place? What kind of stupidity would that be? Are you seriously, seriously suggesting Apple shouldn't fix this?

 

[Deuce on the Clock]

Right. See what the person I quoted was saying...

The implication was you "have to use safari for best battery life" and thereby it's not a pro machine. Of course Apple isn't going to test third party browsers. You don't "have to" use safari. But the numbers they advertise will be based on their safari testing....

Right. See what the person I quoted was saying...

The implication was you "have to use safari for best battery life" and thereby it's not a pro machine. Of course Apple isn't going to test third party browsers. You don't "have to" use safari. But the numbers they advertise will be based on their safari testing....

Yeah gotcha, I didn't understand his comment at all tbh.

 

[moonjelly]

Why Apple even bothers with these CR idiots is beyond me. SJ would have ignored them. They are irrelevant and only make the news when they hate on Apple... big surprise, they find some reason to hate yet again.

Stick to reviewing toilet paper and tacos, it's what you're good at, CR.

I wouldn't trust them to rate tacos. "In our canned tests against canned tacos, your Mexican grandmother's homemade tacos had inconsistent shelf life, PH, and economy. What? Flavor...?"

 

[Naimfan]

Why are people so lazy and dense? Why can't people read and comprehend the written word? Let me spell it out really clearly.

Apple makes a claim that you get X hours browsing on the MBP.

Consumer Reports wants to validate that claim.

Consumer Reports devises a strategy to simulate browsing.
Issue #1: different web sites download at different speeds
Issue #2: I can't have a person sit there and randomly visit websites because who is to say that they will get a random sampling of websites that are fast and slow, simple or complex, etc.
Solution: Simply create a website that has average speed, and average complexity and load it.
Problem. Browsers cache content so after the initial load, refreshing doesn't use much power.
Solution. Turn of caching. Now refreshing a web page of average speed and complexity over and over simulates randomly browsing random sites

TL/DR Consumer reports methodology was just fine, I defy you to come up with an alternative method for making a measure of average power consumption visiting average websites until the computer runs out of power. They did acknowledge that the results are somewhat harder than real life, because obviously some sites we go to are cached. But the results are the same for every machine they test.

What then happened is that using this very logical testing process, Consumer Reports uncovered a bug with how Safari handles icons. This resulted in the same test in Safari using up much more power that the test in Chrome.

If Consumer Reports redoes the test using Safari, they will NOT change the methodology, the claim is that Apple fixed the bug in Safari and now with the cache disabled, the test will work as expected. Nobody is asking CR to change the test for Apple, only to do it on the fixed software.

The only "fault" here lies in the fact that Apple had a bug that was exposed by Consumer Reports very logical testing procedure. AND that Apple did not want to comment until they figured out what was causing the problem.

I don't think Consumer Reports has a responsibility to wait for Apple before posting their results. If anything, they did more than they needed to by letting Apple know what they found.

Who knows what other bugs are waiting for just the right set of circumstances to wreak havoc.

Now I will sit back an watch post after post come that ignores these fact in the name of apple hate or apple love.

Exactly correct. Well said!

 

[coolfactor]

The only "fault" here lies in the fact that Apple had a bug that was exposed by Consumer Reports very logical testing procedure. AND that Apple did not want to comment until they figured out what was causing the problem.

But Consumer Reports portrayed that as a "MacBook Pro" battery issue, not a Safari issue. Big difference.

 

[jayducharme]

Nope. You're the first and only one to suggest this. Congrats?

 

[69Mustang]

You need to read more carefully. CR said themselves that without caching disabled, they had consistently long battery life. With caching disabled, they didn't. With the original, unfixed software. And disabling the cache is something that 99% of users don't do, and the one percent does it rarely. So they got results that were irrelevant to real life usage.

I read it carefully enough. My issue isn't with CR's results per se. It was with your claim of degraded battery life with cache disabled while ignoring the inordinately high battery life experienced with cache disabled... in the same testing cycle. For the record, none of the testing using web pages is indicative of real life. Constantly loading 10 web pages, or in Apple's case 25 pages, is not real world usage. Moreover it's a synthetic test. The goal is to use the same testing procedures on all products. Not to create product specific testing to make a product look good or bad. Remember, disabling cache was never an issue with any of the other CR test with Apple's previous laptops.

 

[canadianreader]

Amazing how this macbook is not well recieved

 

[anzio]

Do you? Why? Other than complaining on MacRumors, what would be your reason?
And do you use a laptop on battery when you do this?

Because I don't want any cached files breaking a new update to content on a server (or even making a new update appear to work when really it's not working with the latest files). I wasn't complaining, I was simply asking how a fix would make its way to me so I can get the optimum battery life. I love this laptop and want to make sure I have all available patches. This is an extremely useful feature, which is why it's so easily accessible in the Develop menu.

Sorry for wanting the latest bug fixes to developer features that may give me greater battery life in the event that I use it. Why the hostility towards any seemingly positive comments about this machine?

 

[The Game 161]

luckily my 2015 MacBook pro is still great.

 

[bryan.cfii]

So I understand that consumer reports was using settings that were a little non standard compared to the "average user" but come on... They were settings that were available in Apple's software, right?

It's not like Consumer reports hacked Safari and jury rigged it to do their tests.
It was a Safari bug.
Okay, but why is this stuff always a finger pointing issue with Apple? Sure, it's an obscure bug that probably <1% will ever encounter, but still.

Consumer reports tested something and found a hole. I don't think it's so much that Consumer Reports "tested it wrong" which feels like this is how Apple is trying to spin this. It may be odd for that niche of that industry, but plausible testing is plausible testing.

Isn't the idea of development to weed out bugs and fix them?

Oops for Apple, but I feel Apple should just take responsibility and say that they never accounted for that, fix it, and resubmit. I feel like an attempt to blame Consumer Reports is being made here with a smile and a subtle way.

I feel Consumer Reports should have been more upfront about their methods so that Apple didn't have to weed that issue out or wonder what the hell is going on, and I feel Apple should fix an obvious issue that if I remember right, would also effect private mode too?

Just my opinion though.

 

[moonjelly]

Exactly, the test made sense.

In fact, Apple does very much the same thing when they do battery tests. (Serving a set number of pre-stored pages over and over again.)

Except apparently Apple doesn't turn off the cache in their tests. Which is interesting.

Does this mean Apple's battery tests are bogus because they're actually hitting the cache all the time?

Or are they using a different method to make sure each page is actually served.



It's not up to CR to figure out a bug like this.

Heck, it looks like it took Apple over a week to figure it out... and it's their code.

The upshot is everything is sorted out, and that's all that matters. Only on an Apple fan site would this rate more than a dozen posts.

Apple testing might have turned off cache but still not triggered the bug. They said it was obscure. Anyone in software can tell you how extremely obscure triggering a bug can be.

Its not Consumer Reports' job to find the bug. It is their job to be informed about the nature of their own canned tests; when the test is behaving inconsistently; what it means to the consumer. Not recognizing the issue was software related and therefore transient and therefore worthy of a footnote in the rating is a failing. They treated software as hardware. Its ignorant and not a service to their subscribers.

I wholeheartedly agree that on this site things have been getting blown out of proportion recently.

 

[TheShadowKnows!]

Why are people so lazy and dense? Why can't people read and comprehend the written word? Let me spell it out really clearly.

Apple makes a claim that you get X hours browsing on the MBP.

Consumer Reports wants to validate that claim.

Consumer Reports devises a strategy to simulate browsing.
Issue #1: different web sites download at different speeds
Issue #2: I can't have a person sit there and randomly visit websites because who is to say that they will get a random sampling of websites that are fast and slow, simple or complex, etc.
Solution: Simply create a website that has average speed, and average complexity and load it.
Problem. Browsers cache content so after the initial load, refreshing doesn't use much power.
Solution. Turn of caching. Now refreshing a web page of average speed and complexity over and over simulates randomly browsing random sites

TL/DR Consumer reports methodology was just fine, I defy you to come up with an alternative method for making a measure of average power consumption visiting average websites until the computer runs out of power. They did acknowledge that the results are somewhat harder than real life, because obviously some sites we go to are cached. But the results are the same for every machine they test.

What then happened is that using this very logical testing process, Consumer Reports uncovered a bug with how Safari handles icons. This resulted in the same test in Safari using up much more power that the test in Chrome.

If Consumer Reports redoes the test using Safari, they will NOT change the methodology, the claim is that Apple fixed the bug in Safari and now with the cache disabled, the test will work as expected. Nobody is asking CR to change the test for Apple, only to do it on the fixed software.

The only "fault" here lies in the fact that Apple had a bug that was exposed by Consumer Reports very logical testing procedure. AND that Apple did not want to comment until they figured out what was causing the problem.

I don't think Consumer Reports has a responsibility to wait for Apple before posting their results. If anything, they did more than they needed to by letting Apple know what they found.

Who knows what other bugs are waiting for just the right set of circumstances to wreak havoc.

Now I will sit back an watch post after post come that ignores these fact in the name of apple hate or apple love.

Total clarity. Absolutely spot on.

[In my opinion, CU uses credible engineering behind all their tests to provide unbiased comparison among brands. Where they may lapse is on their reliability ratings, which are based on statistical synthesis of responses from subscribers to annual questionnaires -- CU does not do long-term product testing, because of obvious economic constraints, and that may be their only fail.]