Researchers Struggle to Use Apple Watch in Studies Due to Sudden Algorithm Changes

[Will Tisdale 🎗]

The FDA has not certified that feature to a high level so it falls under the fun and wellness category. One level above toy. Too many people are giving the watch unwarranted praise and attention and Apple their cash.

I wonder what certification it’s had outside of the US though, and which regulatory agencies have been involved? That kind of information seems to be impossible to find.

 

[FelixDerKater]

Just the facts...

 

[icwhatudidthere]

The concept of API versioning is common knowledge among developers. Apple has already expressed an interest in working with researchers for health research applications. One solution that could be done here is for Apple to add versioning of the data sets that are returned.

 

[0924487]

I'm going to guess Apple didn't make the raw data available to avoid this very case where different applications could interpret the data differently. That at least reduces the number of culprits responsible for sketchy data interpretation to just Apple.

I wonder if they could put something in the EULA to make raw data available for scientific studies along with a vanilla sample algorithm for consuming it. Any studies could then publish the algorithm they settled on -- whether baseline or tweaked -- along with their raw data.

No, there is no need to provide RAW data. As an Econometrician, I think Apple has a very easy solution to this problem. They only need to export processed data along with a post-processor version number. They should allow researchers to choose a specific post-processor version before they export data. As long as this is kept consistent, then there is no need for external parties to get RAW data, which requires subjective interpretations anyway.

 

[Wackery]

You don't have to be a medical expert to understand data analysis.
If the baseline data is changed via an algorithm update, you need to know how that algorithm was changed to account the data discrepancy.
Without the info, the data becomes useless.

You really think Apple doesn’t understand this and just willy nilly changes the algorithm without considering the consequences?

 

[DocMultimedia]

Apple needs to allow access to raw data. It's that simple.

As a physics researcher in my younger days I know you can't use "mysterious" data for your research. If Apple tagged all data with the API they used and allowed you to reverse it to raw data that would be fine, but I'm sure they are using some compression/non-reversible algorithm.

 

[Solver]

Changing the methods in which data is gathered, analyzed and used to obtain results while the same time still historically comparing these results with previous results has become way too common these days. It's almost like only a few care about real science anymore.

 

[jimbobb24]

Naive study design by old researchers. Software needs to be fixed as part of study design and this should be coordinated with Apple. This isn’t a piece of traditional medical equipment that is shipped out and never sees an update. This should have been anticipated and appropriate steps taken for mitigation from day one.

 

[Shirasaki]

This is so incredibly disappointing, and yet also so undeniably Apple.

It's both funny and sad that one of the few aspects of Steve Jobs Apple that Tim Cook has chosen not to change, the company's intense secrecy, continues to be its downfall in so many current and potential markets.

Apple is fortunate to be the only real player in the wearables market at the moment. Despite the complaints, I would imagine most researchers will continue to work with Apple for the time being as they're the only game in town. That said, just as in the pro and corporate markets, Apple will eventually struggle to retain research clients in the long run if it continues to cling to its asinine refusal to provide even a basic level of transparency.

If Apple wants to expand outside of the consumer market in a serious way it needs to understand that secrecy is not the asset it is there but a serious liability that many folks just aren't going to be willing or able to deal with.

Only if Apple is willing to cannibalise its own pride, culture and tradition. Until then, no change will be made even under the law.

 

[vegetassj4]

Wait, Rate Gate due to an update? I’m irate.

 

[jimbobb24]

Lots of people freaking out and reaching weird conclusions about what is essentially an irritation for the biostatisticians working on these studies. But because it involves Apple it makes headlines and people moan.

 

[Shirasaki]

Apple needs to make those algorithm changes available to medical professionals so they can better understand the data. Otherwise the Watch is useless as a tool for measuring accurate data.

I’d argue researchers should give up using Apple Watch for long term research projects. It’s mostly a toy to gamify individual health after all.

Or, researchers try to not update software before the research is done and see if algorithm changes.

 

[Shirasaki]

The FDA has not certified that feature to a high level so it falls under the fun and wellness category. One level above toy. Too many people are giving the watch unwarranted praise and attention and Apple their cash.

Yeah, Apple Watch is basically an expensive toy that gamify individual health and fitness. Nothing more.

 

[dwsolberg]

Apple's heart rate variability and VO max readings are way, way off for me and very inconsistent. For example, my HRV from Apple's automatic reading is around 20, which is very bad. If I use the Breathe app and put my finger on the dial, it's about 120, which is very, very good. Even if there's a reason for the difference, how is a person supposed to interpret that?

 

[Nuno Lopes]

Personally never took the Apple Watch that seriously regarding accuracy. Just as an indicator. This is no medical device, neither is the company supporting it as such. Still when does marketing stops being marketing and becomes a lie? … Gl with this one … read the EULA … there are no assurances … just best effor.

 

[lkalliance]

** never mind, thought better of the post, have a nice day, lol! **

 

[Alfredo_Delgado]

Because of memory constraints, I bet they don't store the actual data stream coming off the sensors. Rather the software on the watch pre-processes the data. Perhaps the Apple Watch has a streaming mode that is accessible to apps on a connected phone. I know some Samsung watches have this.

I wonder if there could be some sort of "research mode" on Apple Watch where research apps could have access to "pure" data that isn't algorithmically distorted.

I think Apple has a very easy solution to this problem. They only need to export processed data along with a post-processor version number. They should allow researchers to choose a specific post-processor version before they export data. As long as this is kept consistent, then there is no need for external parties to get RAW data, which requires subjective interpretations anyway.

If Apple tagged all data with the API they used and allowed you to reverse it to raw data that would be fine, but I'm sure they are using some compression/non-reversible algorithm.

It sounds like the competing reactions are "researches should cope" vs. "Apple should do something" with the latter requiring that the raw data be preserved somewhere.

 

[jerryk]

I worked for 30 years doing computer simulation of petroleum reservoirs. Figuring out the history to match prior to prediction mode was half the work.

Just 1/2 the work.

 

[iPadified]

Yeah, Apple Watch is basically an expensive toy that gamify individual health and fitness. Nothing more.

That is not bad as it increases awareness for health. Apple Watch is sufficient as a screening device that sometimes, but not always, can help finding health issues in an individual.

It is not FDA approved for a reason. If it where, changing the algorithm would not be allowed without a new FDA approval.

Like many instruments AW obviously needs to be calibrated now and again. What did the researchers expect? This is no different from nearly all biomedical instruments I know of so why would AW be different? Furthermore, you do not get the raw data as readings of (usually) electrical signals from ANY biomedical instrument unless you build them yourself. All data is processed. The solution: calibrations and comparing to standards (internal and external).

 

[bluemonkeyguy]

Yeah, the VO2 max calculations are massively off too, and they’ve fiddled around with that algorithm regularly too.

Apple’s calculation is consistently less than two thirds of my actual, and they claim this has been validated, not that they provide any evidence.

I filed a feedback about it around a year ago with detailed information as to how I discovered it was so inaccurate - I’ve had no response.

When the watch updated from a significant version 7.x -> 7.y a while back (I can't remember the exact version), I noticed the ECG app failing to output an average readout consistently. Fidgeting here and there to get a result was frustrating until I realized the ECG algo was modified in that new update. So disappointing; I wish we would be able to go back to previous algorithms. And the data be tagged as such with their specific version.

 

[ksec]

Macrumors comments, Apple should pull out of this University Researching programme. It is Apple's devices and Apple's Data.

 

[randyhudson]

Real researchers would export the raw heart rate measurements, then use their own algorithm to throw out outliers and establish a resting heart rate.

What they failed to recognize though, is even the "algorithm" used to take a single measurement changes between watch OS releases. I used to get readings as low as 40 bpm, but after the last major update it rarely goes below 47.

 

[Jason Zwolak]

This is a common problem in software development: functionA does one thing in v1.0 and another thing in v2.0. What the software development community as a whole should be striving for is stability in the semantics of all functions regardless of version. If a new semantic is desired then a new name should be used. In the most simplistic case it could be functionAv2, but ideally the new name would capture the new semantics. If a developer / project owner wishes to deprecate an old semantic, they can still label it as deprecated, but the function should still remain precisely for cases like the one mentioned in this MacRumors post. And especially for cases like long term research, the algorithms implemented in these functions need to remain accessible in some manner. Apple likes to move forward and cut the fat and bloat of legacy code and I get that... I'm sure they can devise a way to make the algorithms available to just those actively using them in a study for the duration of the study. This wouldn't solve the problem of future studies that may want to compare something to an older study, but at least a particular study could be completed reliably.

Here's a nice talk on the subject of API stability: Coding for Growth: Spec-ulation

 

[Nicky G]

Welcome to working with data. Getting and pre-processing data to improve quality uses upward of 70% of the project time. Data is constantly changing due to legal or technical issues or to make it better for users other than you. Any study based on data needs to be able to handle these changes.

While this is definitely true in some fields, health and medical data are not areas in which this is true. Scientists absolutely require consistency in their data sets, for the data to be meaningful, and the scientific method has safeguards and practices in place to try to ensure this as best as possible.

Apple is clearly the party that is missing the boat here, thinking that "the Apple way" is always the best way, even when it's not, to an absurd degree. Amazing some of Apple's internal folks allowed this to happen, especially those they've hired specifically into the health and medical data team, who come from backgrounds that absolutely know this stuff.

 

[randyhudson]

In order for a study to account for the changes, it needs to know what was changed. It would need the raw data or the algorithms to compare, so they can account for those changes.
Since Apple doesn't provide that information, the data collected is largely useless.

The raw data is absolutely available. Every HR measurement ever taken. The "researchers" are taking shortcuts by using Apple's interpretation of the raw data instead of their own. In addition, they are claiming "Apple changes its algorithms regularly and without warning". This is BS. They were just too lazy to disable auto-updates.