Consumer Reports Retesting MacBook Pro Battery Life After Apple Says Safari Bug to Blame

[macsrcool1234]

Did you think for 5 seconds before you posted this?

Of course this could have been avoided if CR reached out to Apple before rushing to publish. Something is clearly wrong when their test results had such wide variances.

That must be why Apple is fixing a bug, right?

Sounds like you need to take your own advice.

 

[Glideslope]

Surprising? Shouldn't be. As with most issues these days, if accurate, Crappy Code ONCE AGAIN.

 

[leman]

For all those who are complaining that CR was incompetent, dishonest, etc., take note: There was a real problem, and it was a bug in the MBP/Sierra. The fundamental problem was caused by Apple, not CR.

Sorry, but no. First of all, CR explicitly stated that they were getting good battery life with Chrome but this didn't count for some reason because it's not representative. And at the same time they were using Safari in debug mode - which was never intended for a normal user! How is that supposed to be representative?

 

[Rogifan]

What a pathetic response to cover up for Apple and the anointed Chief Shill Phil and an acknowledged bug in Safari.

If I was testing and got such inconsistent results I would want to know if the machine I had was a lemon or if there was a software bug that was causing it. All of those things can be part of a review. Now CR will re-test get good battery life and change their recommendation. I'm sorry but CR has no credibility. The only reason they publisted that was to get headlines.

 

[manhattanboy]

People seem to be mis-understanding the explanation when they say "they were testing it wrong" - it's clear from the explanation that they were simply using Safari wrong...

The bigger story here is that Apple's software appears buggy as hell. They need to do a couple snow leopard like releases where they just focus on cleaning up garbage code instead of pumping out new features.

 

[antonis]

I understand the desire by CR, but that isn't actual usage. If they wanted to mimic how most of us use the machine they'd not turn off the caching. While its good that apple fixed the bug, it seems kind of sketchy for CR to do that

Please define the "most of us" part on a - so called - Pro machine that apple also had the nerve to suggest that it can be used as a workstation (as a pathetic excuse for the 2013 Mac Pro fiasco). Is a video editor in "most of us" target group ? A photographer ? A musician ? A facebook user in Starbucks ? Or a web site designer that will definitely use that said Developers menu options when creating sites ?

 

[Menneisyys2]

So, after all, Safari was indeed buggy. That is, it's not because of the "incompetence" of CR that they originally got the bad results but that of Apple's bug.

 

[maxolson]

Clearly not a Pro machine if you are required to use Safari to get best battery life.
The older MBP laptops could use any browser, or amazingly it it could run Pro tools like Adobe products without killing the battery. Apple, if you say it has up to 10 hours for a Pro laptop, then we expect around 8 or 9, not 2 or 3. Safari is no excuse.

EDIT: Could we stop ranting about how Chrome is better. I'm just trying to point out that Apple is no longer designing this laptop to have good power for Pro software like Adobe products. Chrome I would not consider a Pro product.

Dude, you clearly didn't read the article.

 

[69Mustang]

I understand their motivation. That's not a a real life usage test, though.

And this is? "Testing conducted by Apple in October 2016 using preproduction 2.6GHz quad-core Intel Core i7-based 15-inch MacBook Pro systems with a 256GB SSD and 16GB of RAM. The wireless web test measures battery life by wirelessly browsing 25 popular websites with display brightness set to 12 clicks from bottom or 75%.

Real world usage involves more than just repeatedly browsing 25 cached websites.

That's not saying there's anything wrong with the way Apple tests. Far from it. CR is not Apple. CR has to test across a wide array of products. Testing across that array means getting as many criteria as possible to be the same.

 

[diddl14]

What a surprise, just a software bug...

Been using a 15" for weeks now and it's just perfect.
Zero battery issues, super fast and super slick.

 

[dxmac99]

so the 10-hour battery life for web surfing is only good for visiting the same website within 10 hours. I start to wonder what kind of genius Tim Cook has been hiring lately....................

 

[Rogifan]

That must be why Apple is fixing a bug, right?

Sounds like you need to take your own advice.

And had CR reached out to Apple it would have been discovered before they published. Again it wasn't that the battery results were bad it's that they were incredibly inconsistent. Wouldn't normal course of business cause someone to analyze results before publishing them? And if the results are that abnormal perhaps do a bit more analysis before publishing? Seems clear to me CR rushed to publish this to make headlines. Every tech site out there reported on their findings.

 

[macfacts]

Consumer Reports sucks. The fact that they say they turn off caching to work the batteries harder, but are also trying to show consumers what they can expect is a classic red herring. I would bet significant money that the typical Consumer Reports reader would NEVER think to turn off page caching in Safari, let alone actually know how to do it, so CR doing so was setting up an unrealistic / uncommon scenario for what people could expect.

Apple should sue them, to recover the $18,000.00 in lost revenues caused by the 10 CR readers who bought a PC instead of a new rMBP.

Turning off cache was to simulate a person browsing different sites. Most people don't sit around hitting F5 on one page.

Most importantly, the inconsistent battery life was because of a bug in that setting, not because that setting was used.

 

[Gerlandano]

Clearly not a Pro machine if you are required to use Safari to get best battery life.
The older MBP laptops could use any browser, or amazingly it it could run Pro tools like Adobe products without killing the battery. Apple, if you say it has up to 10 hours for a Pro laptop, then we expect around 8 or 9, not 2 or 3. Safari is no excuse.

EDIT: Could we stop ranting about how Chrome is better. I'm just trying to point out that Apple is no longer designing this laptop to have good power for Pro software like Adobe products. Chrome I would not consider a Pro product.

You need to read more carefully. You are not required to use Safari to get the best battery life. It actually got better battery life with browsers OTHER than Safari. The whole problem turned out to be a bug in Safari when using it in "Developer Mode" which Consumer reports was doing. Apple has now fixed this bug in in Safari. The laptops do get the battery life as advertised. I use one with the Adobe Pro Apps, Davinci Resolve, and other Pro apps and it works longer on batteries than any other Apple laptop I have ever used.

 

[mw360]

Why are so many people taken in by Apple hinting that the test settings were somehow odd, obscure or rigged? Turning off the browser cache isn't particularly arcane and makes perfect sense to me if your test involves loading the same pages over and over (which it should, to generate fair comparisons between models). Fetching webpages entirely from the cache over and over isn't exactly real world usage either.

 

[FatherJack1980]

If you don't tweak settings to make tested products fairly compete head-to-head, there's no real testing results worth anything. For example, all it would take in a "default" settings approach is for one tested laptop to turn screen brightness down to zero. CR makes it an apples to apples comparison so that their ratings are reasonably objective in comparing products for us consumers.

Again, most of the Apple hardware rated by CR in the past has been rated very highly. They are not dynamically changing their testing approaches from product to product to single this one laptop iteration out.

I'm not a US citizen and hadn't heard of Consumer Reports before; but from the previous article here on Macrumors, I didn't read anything how they are testing one product and then comparing it to all others; what my understanding was from that previous article was that they test products to see if it lives up to the claims of whoever manufactures them

 

[TMRJIJ]

Clearly not a Pro machine if you are required to use Safari to get best battery life.
The older MBP laptops could use any browser, or amazingly it it could run Pro tools like Adobe products without killing the battery. Apple, if you say it has up to 10 hours for a Pro laptop, then we expect around 8 or 9, not 2 or 3. Safari is no excuse.

EDIT: Could we stop ranting about how Chrome is better. I'm just trying to point out that Apple is no longer designing this laptop to have good power for Pro software like Adobe products. Chrome I would not consider a Pro product.

Still didn't read the article though..

 

[cracka]

I'm glad they ID'd the source of the wildly variable results, but I'm still thankful that the Apple extended holiday return policy allowed me to return my MBP 15 this past weekend.

My daily use is modest (web browsers, WiFi, and text editors), and my results were extremely consistent: ~4.5 hours of battery life. My 2013 retina MBP 15 was consistently good for 7-8 hours performing the exact same tasks using the same software.

It's beautiful hardware, but it just didn't meet my needs, nor did it come close to measuring up to the marketing spiel. I'll stay tuned for the next iteration.

 

[manu chao]

But 3 hours battery life when using one music pro application, for the princely sum of £3.5k?

In the 15" version, the previous MBP had a battery that was 31% larger. Everything else equal, that same pro music application would have lasted a bit less than 4 hours, in a computer rated for 9 hours. I don't remember people walking around with pitchforks back then.

The real problem is not the exact number but that things got worse in quite a number of use cases in a noticeable way. They also got better in some use cases but even if the average battery life over all users stayed the same, worse for some, better for others, the voices of those at the loosing end will always be much louder than of those at the winning end.

 

[MH01]

If I was testing and got such inconsistent results I would want to know if the machine I had was a lemon or if there was a software bug that was causing it. All of those things can be part of a review. Now CR will re-test get good battery life and change their recommendation. I'm sorry but CR has no credibility. The only reason they publisted that was to get headlines.

I'm not sure how many would trust you with "testing" testing involves being impartial and objective.

 

[ahostmadsen]

I don't really get the bashing of CR. CR is the only really independent tester in the US. The Verge etc. get models from Apple ahead of time. If they publish a scathing review, do you think they get a model from Apple ahead of time next time?

CR doesn't work with the manufacturers. That's why they published their review before contacting Apple. That's part of being independent. And their poor results were due to an Apple bug.

I don't always agree with CR, their reviews tend to be somewhat robotic, and don't rely much on them for products that I know about (e.g., electronics). But I do rely on them for products I know nothing about: appliances, grills, lawnmowers etc.

 

[HobeSoundDarryl]

If I was testing and got such inconsistent results I would want to know if the machine I had was a lemon or if there was a software bug that was causing it. All of those things can be part of a review. Now CR will re-test get good battery life and change their recommendation. I'm sorry but CR has no credibility. The only reason they publisted that was to get headlines.

So they have no credibility when they rate most other Apple tech so highly too? Or no credibility because they found some fault with the ONE Apple product this one time?

CR's role in the world is OBJECTIVE product reviews. They aim to help us consumers make good decisions. They rank many products from many manufacturers very poorly too. This includes products from companies like Samsung. Presumably, since they have "no credibility," we should not trust their negative reviews of Samsung products too?

Since they rate so much stuff, I doubt that headline chasing is high on their list of activities. Else, why rate all kinds of stuff that only niche segments of consumers might care about. Why not focus on only the most "in the news" items? They don't sell ads, so they are not chasing eyeballs to see ads. They don't even present themselves as a business (they are an ORG, not a COM).

When I'm buying something they rate, I check their ratings... and trust them. Why? Because they have no agenda, no bias to one manufacturer over another, no financial incentives to make one product rate poorly or another rate highly. Where else can we consumers get such objective reviews done by professional reviewers? Here? Amazon reviews? Any site with an ad revenue model?

 

[69Mustang]

And had CR reached out to Apple it would have been discovered before they published. Again it wasn't that the battery results were bad it's that they were incredibly inconsistent. Wouldn't normal course of business cause someone to analyze results before publishing them? And if the results are that abnormal perhaps do a bit more analysis before publishing? Seems clear to me CR rushed to publish this to make headlines. Every tech site out there reported on their findings.

You keep saying this. Rush to publish. What's their end game? Also, is this how you would have handled any failed test? Say, if you were testing the S7 Active maybe. Samsung claims CR had defective units. Should they have not published those results?

 

[brentsg]

I stand by what I wrote. It seems a crowd of us is trying to make this about a testing protocol. It's not about that at all.

You posted a boatload of stuff unrelated to the comment I made. You said "No, this just altered the standardized testing for all laptops to improve Apple's results."

It isn't about that. It's about a bug that needed to be fixed. The testing protocol uncovered a bug, and that bug lead to poor battery life. End of story.

It is NOT about Apple wanting to be tested differently than everyone else, though I'm sure that all manufacturers would like the testing protocol changed because it would improve everyone's results, not just Apple.

Consumer Reports needs clicks. They need to post the most terrible, ridicule inducing reports they possibly can because that controversy leads to page clicks. That's what motivates them. I used to rely on them for things like dishwashers, but I've since found that to be a mistake as well. Best to just talk with people that repair all of them.

Regarding the publishing of results, it's VERY common in the industry to reach out to manufacturers when it seems something peculiar has been identified and can be reproduced. This happens all the time in computer product testing, and even for other types of products. Sometimes the manufacturers can investigate, and other times they can provide another sample of the product in case it's a simple defect.

 

[NT1440]

The bigger story here is that Apple's software appears buggy as hell. They need to do a couple snow leopard like releases where they just focus on cleaning up garbage code instead of pumping out new features.

....How does a hidden system feature that end users will never enable point to "buggy as hell" software?