Consumer Reports Retesting MacBook Pro Battery Life After Apple Says Safari Bug to Blame

[ArcaneDevice]

Do people still turn to Consumer Reports? It's certainly not a website or magazine I think of when looking to buy a MacBook Pro.

With the plethora of popular tech sites out there, I'm surprised Apple paid much attention to them. That said, the executives at Apple all probably grew up reading CR.

Given the user base of Apple is as generic and mainstream as it gets now Consumer Reports is perfectly at home testing Apple products next to refrigerators and is still a well known brand and central focus for product reviews.

You think the majority of Apple customers are going on the internet checking out Geekbench and getting their reviews from Ars?

 

[RedOrchestra]

It's been used by several Apple folk; maybe it's part of the onboarding process to state things like this.

Hey Eddy, that's the best untucked shirt you've ever worn.

 

[crazylegsmurphy]

Apple says it isn't a setting used by most users. So, no web developers ever use Macbook Pro computers?

If it was indeed a bug, then the only response from Apple should have been, "We thank CR for testing our products so thoroughly. This high standard of testing has revealed a bug in Safari that would have affected many of our pro users. We have addressed this bug, and invite CR to retest our product, which we are confident will pass or exceed thier standards."

 

[HobeSoundDarryl]

I have to think that if this was Microsoft this wouldn't be happening.

If this was Microsoft (Surface hardware), the collective sentiment would be to rip the hardware to shreds on the strength & confidence of an objective CR review. "Our" problem here is that it became a CR vs. Apple issue, so CR had to be wrong. If it was God vs. Apple, God would have to be wrong.

However, swap it out so that it is CR faulting Samsung, Microsoft, Google, etc and we'd be rah-rahing the CR review like it was written by God himself.

 

[citysnaps]

It still doesn't fix the problem with the 2016 model having 1/3 smaller battery than the 2015 model.

How is that a problem if it delivers the advertised battery life?

 

[Jaspers dog]

Do you really care about the battery life for whatever your pro work?
If you are working on anything intensive, you have to plug the power anyway.

Herein lies the divide in opinion on this. I'm a professional photographer and frequently don't have access to power on location, that's why I'm using my Macbook Pro and not my Mac Pro (that and the fact that it's not practical in the middle of a field for a number of other reasons) Apple make 3 products already for people who priorities are size and weight but not power. Many of us value power and battery above those considerations so this release is a disappointment. Sure I like new form factor but I didn't need it to be slimmer per se and neither did many "pros". Also we don't have any choice but to use 3rd party apps because Apple don't make the software we need.

 

[citysnaps]

The classic Apple "You were testing it wrong" ploy

You didn't fully read (or comprehend) the article, did you?

 

[nofear1az]

So why wouldn't any of the other MacBooks including the older ones show the same battery disparity results as this new model does with this test? Were they ever tested? Something is odd here.

It still seems if you tag the system with real work the battery drains faster than it should but if you don't tag it it gets really good battery life.

 

[Return Zero]

Some people can't read...

According to Apple, it wasn't the caching setting itself that caused poor battery life, it was that the setting was triggering a bug which caused icons to constantly be reloading for no reason.

According to Apple, this bug is now fixed in the latest beta release. One would expect a new CR test that once again turns off caching to yield much better results this time.

 

[DaPizzaMan]

Why would CR disable caching when browsing to perform their test? By default, people leave caching on. Unless they're trying to find the worst case with battery life.

Also, yes, I know that the issue was not that they disabled cache but a bug with that function being disabled.

 

[HobeSoundDarryl]

But boy, I would have thought Consumer Reports would have default settings on everything on all laptops they test (just as about every John Doe that buys a new laptop - not that many tinker around with settings). Now it makes more sense

If you don't tweak settings to make tested products fairly compete head-to-head, there's no real testing results worth anything. For example, all it would take in a "default" settings approach is for one tested laptop to turn screen brightness down to zero. CR makes it an apples to apples comparison so that their ratings are reasonably objective in comparing products for us consumers.

Again, most of the Apple hardware rated by CR in the past has been rated very highly. They are not dynamically changing their testing approaches from product to product to single this one laptop iteration out.

 

[triton100]

How is that a problem if it delivers the advertised battery life?

It doesn't deliver the advertised battery life by a large margin.

I have run copious real usage tests. It under performs advertised life by around 30 percent and has 10 percent less battery life than then 2015 model.

 

[justperry]

No, this just altered the standardized testing for all laptops to improve Apple's results. Basically this is "you're holding it wrong" again. With that phone, holding it differently yielded better results. With this testing, changing a common testing parameter used with all laptops is going to yield a much better result.../ snip

Seems like you didn't read the article, it said Apple found a bug which they have now fixed.

Apple has since concluded its work and learned that Consumer Reports was using a "hidden Safari setting" which trigged an "obscure and intermittent bug" that led to inconsistent battery life results.

 

[Juan007]

Why Apple even bothers with these CR idiots is beyond me. SJ would have ignored them. They are irrelevant and only make the news when they hate on Apple... big surprise, they find some reason to hate yet again.

Stick to reviewing toilet paper and tacos, it's what you're good at, CR.

 

[mauka]

I understand the desire by CR, but that isn't actual usage. If they wanted to mimic how most of us use the machine they'd not turn off the caching. While its good that apple fixed the bug, it seems kind of sketchy for CR to do that

This was a battery test that used repeated full web page loads as the method expending the battery charge. Turning off caching is the correct way to run the test. Nothing "sketchy" about it.

 

[tzm41]

For a company called "Consumer Reports" they weren't testing products in the way Consumers would.

I don't know but this whole thing smells like a goofup on CR end.

Sure there might be a but in Safari, but it's kind of like testing a car and saying "it doesn't get 0-60 in 10s, it doesn't move at all!" and then later finding out that they removed the wheels prior to testing. Consumers wouldn't do that.

Testing methodology was the blame here, maybe 5% of that blame should go to Apple for a Safari glitch but still, that's weak.

Consumers don't load the same site thousands of times (from the cache) until the battery dies, which is why CR turned off the cache so that the browser will reload from server every time to mimic user behavior that they browse different websites. The same setting has been used to test other laptops and previous MacBooks, according to CR. So why would you think it is CR's fault when Safari screwed up?
Last time I checked, VW cannot blame user who drives fast for their diesel engine emission...

 

[nofear1az]

I'm no Apple apologist, but the Consumer Reports test was laughably amateurish, as even the author admitted in the article that Safari could be to blame.

amateurish? how do you figure that?

When you test battery life using the Web, you test it by loading new pages rather than pulling from cached pages. It makes total sense to turn off cache. Sure they could have just had it pull 5000 different web sites instead but why not just turn off the cache mode.

 

[usarioclave]

It's funny, Safari has had historical problems with favicons. Maybe disabling bug was the disabling the cache screwed up favicon handling?

 

[Naimfan]

Why would CR disable caching when browsing to perform their test?

To provide consistency across ALL laptops that they test.

 

[TomMcIn]

I like the comments in these forums that directly discuss the problem.
I have absolutely no use for the TROLL comments that twist and spin the content to try and be misleading or funny. Others just display their lack of ability to read and understand the facts presented. They have already made up their minds and no amount of truth or logic will change their beliefs. If the TROLLS would just keep their wrong beliefs to themselves,.....

 

[jmgregory1]

So CR tested using settings that are available to all users, yet the MBP defenders are saying they did it wrong? But Apple comes along and finds a previously unknown bug, lets the world know that CR was right, and folks are piling on CR?

Head shake, you can't have it both ways folks.

I can 100% guarantee that most if not every CR reader out there is not going to turn off page caching through Apple's Safari Developer settings. And further, just from your own standpoint, tell me how many times are you doing nothing but visiting different websites, one after the other, never re-visiting a single site?

What CR did is not unlike the hardware tests that some people tout as showing how great a PC or Android product is because it hits certain scores for a process that isn't really representative of real world usage. I'd rather see CR test by having people use a product as they normally would during a day, then simply note that your results may vary - which of course is absolutely how it is.

 

[citysnaps]

It doesn't deliver the advertised battery life by a large margin.

I have run copious real usage tests. It under performs advertised life by around 30 percent and has 10 percent less battery life than then 2015 model.

And others who have tested their laptop under the conditions Apple specified, have.

 

[Tubamajuba]

You didn't fully read (or comprehend) the article, did you?

Some people can't read...

According to Apple, it wasn't the caching setting itself that caused poor battery life, it was that the setting was triggering a bug which caused icons to constantly be reloading for no reason.

According to Apple, this bug is now fixed in the latest beta release. One would expect a new CR test that once again turns off caching to yield much better results this time.

Seems like you didn't read the article, it said Apple found a bug which they have now fixed.

Why should they waste their time reading anything? They already know that whatever happens is always Apple's fault, and nothing could ever change their minds. Any new information either proves them right or is biased towards Apple and wrong.

So yes, Apple's bug, Apple's fault, always has been and always will be. If you disagree, you're a shill.

 

[citysnaps]

Yeah, I completely agree. It seems really odd for them to say that they aim to test it under the same type of conditions a consumer would experience, then go out of their way to enable this hidden setting because they randomly want the browser to pull each page refresh from the server. If that isn't a setting most people will ever enable, and thus not how most consumers will use the laptop, it's pretty arbitrary.

For sure. Is my next door neighbor, who is a school teacher, going to 1) Turn on Developer settings, and then 2) Go to the setting to Disable Caches?

Why?

 

[macfacts]

Well, it doesn't really speak for Consumer Reports that they didn't do additional testing. If it's a Safari setting, then an alternate battery life test with Chrome (very common use case) would've easily showed the anomaly in battery life. Why do they have so much credit anyway?

Except CR did test with Chrome in the first report. Don't just listen to Phil.