Researchers Struggle to Use Apple Watch in Studies Due to Sudden Algorithm Changes

[djlythium]

Doesn't surprise me. I have an S6 that I've stopped having collect data on me because the biometrics aren't really that accurate, or otherwise suffer from concerning variance. (source)(source)(source)(source)(source) These are just some of the most recent empirical studies on wearables' validity and reliability.

I feel like I tried to be cognizant of the Watch's limitations, looking up studies before I bought one, but when my Oncologist recommended that I don't rely on it, I looked even more into it. If the field of medicine as a whole is not promoting these devices, I'm not convinced we should trust the data, not even for baseline measurements (because that baseline could be substantially off).

 

[Nicky G]

And if there is a huge change to an algorithm, I need to be able to let the consumers of that data know what changed, why it changed, and how to interpret the differences.

Yeah, see -- we're not talking about consumers. We're talking about medical researchers following scientific research best practices. Constantly changing data sets coming from impenetrable black boxes isn't how it flies, with medical research. Your personal experience with consumer and commercial applications is simply not terribly relevant.

 

[johnnygee]

That does it! I hain't a-buyin' me none o' these here Apple Wortches...Oh wait! I already own one. 🤣

 

[rpmurray]

You really think Apple doesn’t understand this and just willy nilly changes the algorithm without considering the consequences?

Yes.

 

[Razorpit]

It sounds like you have made sure your data is for useful in studies.

As pointed out in the last couple of paragraphs, researchers saw all the things the Apple watch could do and made some assumptions about data fidelity that are simply not true.

Ya, they assumed the data they were receiving was accurate. 😂

It‘s not like these researchers did all the work on their own. Apple created their own research kit; https://apps.apple.com/us/app/apple-research/id1463884356. If there’s a clause in there that says “data subject to change” then shame on the researchers for not picking that up.

 

[centauratlas]

It's the same principle as with raw format in photos – pro users need raw data.

You are exactly right. It is the same requirement for all data - weather, climate, hurricane, you name it. When you have "adjusted data" it becomes worthless because you never know the precise adjustments at the particular time.

 

[ikramerica]

It’s Apple’s fault the researchers didn’t lock their measuring devices down to a specific OS version throughout their study? Allowing for software upgrades during a study without knowing how that will impact your data should disqualify your data anyway.

 

[ikramerica]

No, there is no need to provide RAW data. As an Econometrician, I think Apple has a very easy solution to this problem. They only need to export processed data along with a post-processor version number. They should allow researchers to choose a specific post-processor version before they export data. As long as this is kept consistent, then there is no need for external parties to get RAW data, which requires subjective interpretations anyway.

I enjoyed my Econometrics classes in college but didn’t know what you pros called yourself. Learn something knew.

 

[mannyvel]

One data set is bad, and suddenly everything Apple does sucks. Get some perspective, people.

The question is, "how did the data change?" If they have original copies of the data they can figure out how the new export changed.

The problem is that the old data may have been wrong or inaccurate. That's why you baseline stuff, so that you can figure out how far off measurement #2 is.

Also, it could be a bug in their code. They are academics, and they make mistakes like everyone else.

 

[ikramerica]

One data set is bad, and suddenly everything Apple does sucks. Get some perspective, people.

The question is, "how did the data change?" If they have original copies of the data they can figure out how the new export changed.

The problem is that the old data may have been wrong or inaccurate. That's why you baseline stuff, so that you can figure out how far off measurement #2 is.

Also, it could be a bug in their code. They are academics, and they make mistakes like everyone else.

It’s not just academics.

I good illustration is in the movie office space. Michael Bolton is fired “unjustifiably.” He seeks revenge. So he helps write the money stealing code, in which he makes a “little mistake” that turns into a major FUBAR event, and just brushes it off as little, and then blames someone else. Which kinda explains why he was fired in the first place.

seems as if the researchers and apple are both guilty of this kind of little failure.

 

[Malus120]

Only if Apple is willing to cannibalise its own pride, culture and tradition. Until then, no change will be made even under the law.

I disagree (and what do you mean by "even under the law???").
Apple under Tim Cook has already demonstrated a brazen willingness to throw tradition (the way Jobs would've run things) to the wayside when it suits them.

Furthermore, researchers and to a lesser degree corporate/pro customers aren't asking Apple to open up the magic box for consumers. Apple can still be "secretive," they just need to provide the basic transparency necessary for these people to do their jobs (collect adequately consistent data in the case of researchers and the AW), make business plans/large purchasing decisions (corporate customers), and stake their livelihood on Apple's software (pros.)

 

[Shirasaki]

I disagree (and what do you mean by "even under the law???").
Apple under Tim Cook has already demonstrated a brazen willingness to throw tradition (the way Jobs would've run things) to the wayside when it suits them.

Furthermore, researchers and to a lesser degree corporate/pro customers aren't asking Apple to open up the magic box for consumers. Apple can still be "secretive," they just need to provide the basic transparency necessary for these people to do their jobs (collect adequately consistent data in the case of researchers and the AW), make business plans/large purchasing decisions (corporate customers), and stake their livelihood on Apple's software (pros.)

I’d count secrecy as part of their tradition and they are unwilling to cannibalise that.

As for “under the law”, I’m mocking apple being able to do literally anything they want without consequences, in USA.

They can provide basic transparency, under NDA or not. But then, they should also not to interfere with how research is done. Secrecy sometimes just can’t work right in science field when openness and exchange is crucial. This is more of a mindset shift than a technical one Imo.

 

[jicon]

So, I've used two invasive devices to record interstitial glucose levels... Abbott Freestyle Libre, and Dexcom G6. These two devices somehow have passed certification, meaning their values can be a certain margin away from a true reading.

These are the best devices available, but sometimes laughably bad and inaccurate. The Libre somewhat worse, because once the bad readings come... you're stuck with them. Fortunately the Dexcom offers a 'calibration' option in software.

A device on my wrist that could sit firm, loose, wet, dry, hot, cold... I'm not surprised there are variations from day to night, from person to person... but if the software is updated to recalculate using a different method, at least tell people downloading the update to keep that in mind... and a calibration option would go a long way.

 

[citysnaps]

Apple needs to make those algorithm changes available to medical professionals so they can better understand the data. Otherwise the Watch is useless as a tool for measuring accurate data.

I suspect Apple already does make changes available to medical professionals that formally partner and collaborate with Apple for different studies. Stanford Medicine comes to mind.

 

[txscott]

I don't think it was ever Apple's intent to provide researchers with a data collection tool. I'm pretty sure their goal was to sell watches. Their goal has been achieved.

 

[Mousse]

Apple needs to make those algorithm changes available to medical professionals so they can better understand the data. Otherwise the Watch is useless as a tool for measuring accurate data.

I disagree. The researchers need to choose a different, dedicated device to monitor heart rate.

The Apple Watch is a multi purpose tool, which happens to have a heart rate monitor. It's good enough for the casual user. But when you absolutely, positively have to be spot on, you go with a one trick pony. Those multi-tool (pliers, knife, screwdriver, can opener and so on) makes a great gift for the handyman in your life; I have several. A contractor might use one of those multi-tool gadgets in a pinch, but would be much better served using a dedicated plier, screwdriver, saw, utility knife in most cases.

 

[One2Grift]

I'll call this a built-in feature. Since the data changes each time it's exported, it makes it harder for them to track you. This is Apple putting privacy above accuracy. 🤣

Well then you can switch to the other option right over -------> Android.
Where you get full benefits and credit of getting neither.

 

[One2Grift]

I disagree. The researchers need to choose a different, dedicated device to monitor heart rate.

The Apple Watch is a multi purpose tool, which happens to have a heart rate monitor. It's good enough for the casual user. But when you absolutely, positively have to be spot on, you go with a one trick pony. Those multi-tool (pliers, knife, screwdriver, can opener and so on) makes a great gift for the handyman in your life; I have several. A contractor might use one of those multi-tool gadgets in a pinch, but would be much better served using a dedicated plier, screwdriver, saw, utility knife in most cases.

Stop with that sense stuff! We all know commercial watches are exactly the kind of data critical medical research needs. Forget about those silly controlled environments, Apple is very bad! for updating their commercial device.

It is earnings time. The blather from those looking for free media play is entertaining to watch (pardon the pun).

 

[Dremmel]

Did the same guy who did the app store search algorithm do this one? If so you might want to have a word with him.

 

[thasan]

There are always new processing and computational algorithms to interpret the data better. It’s common sense as the wearable devices do not have very powerful hardware. These sort of devices are everywhere and it is normal to expect gradual improvement of the code that makes the hardware better. I am amazed to see the reaction of people here. These same bunch of people would jump in joy when software tweaks would make their camera get a better bokeh ☺️

 

[Datalinks]

Yeah but their wristbands have been backwards compatible like for ever

 

[Makosuke]

I'm a bit unclear on something from this: It sounds to me like Apple is storing raw data and running all exported data through whatever algorithm is in place at the time of export, not just data collected after the algorithm was changed.

Anyone doing a scientific or medical study would of course vastly prefer the raw data, and it would actually benefit Apple to provide it to them, since they might come up with a better algorithm than what Apple is using, which Apple could then adopt.

...but, since all data is processed through the same algorithm, it doesn't seem like it would render every study useless so long as the study didn't explicitly rely on the old algorithm in its assumptions, the new algorithm doesn't filter out some thing that would have been valuable, and the researchers export all data, not just new data, prior to their final analysis. Those assumptions won't be true for every study, but they certainly could apply to many, and in some cases the new algorithm (if better at filtering out noise, say) might actually improve the results of the study.

That is, let's say I'm trying to correlate heart rate variability to COVID infection. If I export the entire study's data the day before an algorithm change and find a correlation, then when I export all the data the day after that change I should still see the correlation as long as the new algorithm is still accurately measuring.

It might not--the new algorithm might be doing too much filtering so the small changes I was looking for are lost, for example. Or it might actually improve results--if it reduces noise, then smaller changes might be more visible in the statistics. You of course have no way of telling which way the changes went, but then since the algorithm was a black box to start with you can't really say it's made things worse rather than better for your study at the end.

All of which is to say that while Apple should be providing raw data to scientists to get the best value for both humanity and themselves as a company, the act of changing the algorithm through which all data is processed will not necessarily render every study useless, as long as all data is processed through the same algorithm.

 

[4tune8chance]

Welcome to working with data. Getting and pre-processing data to improve quality uses upward of 70% of the project time. Data is constantly changing due to legal or technical issues or to make it better for users other than you. Any study based on data needs to be able to handle these changes.

Yep.
The soon to be released, “Having fun with databases” is the must read for all beancounters who consistently under budget the ……… oh who am I kidding.

 

[rjohnstone]

Yeah, see -- we're not talking about consumers. We're talking about medical researchers following scientific research best practices. Constantly changing data sets coming from impenetrable black boxes isn't how it flies, with medical research. Your personal experience with consumer and commercial applications is simply not terribly relevant.

Context.... reread what I wrote. "consumers of that data"
No where in that sentence did I say anything about consumers as it relates to the general public. I'm talking about the professionals I work with who use the data my systems process. They are the consumers of that data. It needs to be high quality and as accurate as possible.
Everything from data collection standards, ETL processes, and data structure must be reviewed and tested for any variances or changes in methodology used to collect and process the data.
One screw up can have huge financial impacts.

 

[rjohnstone]

I disagree. The researchers need to choose a different, dedicated device to monitor heart rate.

The Apple Watch is a multi purpose tool, which happens to have a heart rate monitor. It's good enough for the casual user. But when you absolutely, positively have to be spot on, you go with a one trick pony. Those multi-tool (pliers, knife, screwdriver, can opener and so on) makes a great gift for the handyman in your life; I have several. A contractor might use one of those multi-tool gadgets in a pinch, but would be much better served using a dedicated plier, screwdriver, saw, utility knife in most cases.

Oh I totally agree with regards to the Apple Watch, and any other consumer grade device. Use the right tool for the job.

Apple needs to really stop with the "Hey look, we got FDA approval for such and such feature!!!" or touting how orgs like WHO use their devices for research purposes. It's all marketing fluff, but man people sure eat that stuff up.