The data is only a guide. No doctor will take those like a gospel.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Researchers Struggle to Use Apple Watch in Studies Due to Sudden Algorithm Changes
- Thread starter MacRumors
- Start date
- Sort by reaction score
I would suspect the raw data also changing. One by different sensors with different software. In order for a sensor to collect raw data, software needed. The researchers should have already know these variables existed. If not one has to wonder just how good a researcher they are. For example, researchers use blood pressure studies by collecting large amounts of data from many different blood pressure devices. In order to get the widest number of samples from the population. All those different blood pressure devices are calibrated differently. I have had, on many occasions in my doctors office, the nurses change the cups or machines and take another reading. Something Not right with this machine. Does the Apple Watch need to be perfectly accurate to get a good study, no. The trends and massive numbers will give the knowledgeable researchers an excellent picture of the data.
Last edited:
People can and do work with changes in data source definitions, resolutions, and formats all the time.
Switching to a dedicated device would be the best thing in your use case. An Apple watch is trying to meet a lot of needs. And some of those may be in opposition to each other.
Apple needs to make those algorithm changes available to medical professionals so they can better understand the data. Otherwise the Watch is useless as a tool for measuring accurate data.
Yes.. I know... it's my day job and I get provided the raw data and the algorithms every time there is a change.
It's my job to make sure reporting data is not only accurate, but consistent.
And if there is a huge change to an algorithm, I need to be able to let the consumers of that data know what changed, why it changed, and how to interpret the differences.
The watch is a tool for consumer use. This is what Apple sells. And it’s pretty useful.
And I assume “more accurate data” is why algorithm was changed. Otherwise why change it? 🤷🏻♂️
Fortunately I never bought into any of the hype about Apple's health monitoring and the 
Watch. Another bullet dodged.
Watch. Another bullet dodged.It sounds like you have made sure your data is for useful in studies.
As pointed out in the last couple of paragraphs, researchers saw all the things the Apple watch could do and made some assumptions about data fidelity that are simply not true.
You're missing the point. "more accurate" compared to what?
Apple LOVES to talk about medical studies that use the Watch as a tool.
The studies become useless when Apple changes how the tool measures data, but then doesn't provide that info to the researchers.
Compared to BEFORE. You think Apple should stop fine-tuning their algorithm? You don’t want your watch to be improved?
This sure is going to make it hard for Apple to get the watch certified as a medical device.
Researchers are rethinking plans to use the Apple Watch in studies after finding inconsistencies in the data gathered by the device (via The Verge).
Inconsistencies in the Apple Watch's heart rate variability data are reportedly caused by Apple tweaking the device's algorithms, meaning that data from the same time period can change without warning. Associate professor of biostatistics at the Harvard T.H. Chan School of Public Health and developer of the Beiwe data platform, J.P. Onnela, told The Verge:
Research fellows at Brigham and Women's Hospital exported the heart rate data from Apple Watches for the same period of time, but several months apart. The data should have been identical, but since it is filtered through an algorithm prior to export it was drastically different.
Apple changes its algorithms regularly and without warning, so exporting the same data at different times may use different algorithms. Beyond heart rate variability, researchers looking into sleep tracking have experienced similar problems with changes to algorithms.
This mostly precludes commercial devices from being used by researchers, resulting in the need for devices specifically designed to collect data for scientific studies. While this is adequate for some studies, it heavily constrains research into the medical value of commercially available products for users. Some researchers say that Apple should publicize the changes it makes to algorithms or make the Apple Watch's raw data available.
Article Link: Researchers Struggle to Use Apple Watch in Studies Due to Sudden Algorithm Changes
The changes don't happen randomly. They correlate to OS updates. I feel like you shouldn't trust hardware / software updates if you want your data to be consistent. Literally anything can change. Also find it weird that they wouldn't mention they are using different OS versions. I've worked with people who use the same data for a research based app on mental health and they talk about the same thing happening between OS updates and having to adjust based on new values of later OS versions.
Without knowing what Apple's rationale is for this (and what constraints they have to live with), I wonder if there could be some sort of "research mode" on Apple Watch where research apps could have access to "pure" data that isn't algorithmically distorted.
Again... no one is complaining about making a device more accurate.
When it comes to analyzing data, especially in long term studies, you NEED consistent measurements with a solid baseline for how the data is collected.
If there is a change to the collection process, you need to know what changed to account for the discrepancies.
Without that piece, the data is useless.
Apple announces three groundbreaking health studies
Three unprecedented medical studies, in partnership with leading institutions, will reach more participants than has ever been possible.
www.apple.com
These 3 studies that Apple bragged about are going to have inconsistent data measurements. That's just a fact given how Apple updates algorithms with system updates.
Unless Apple is providing the algorithm changes (or the raw data) to the researchers, they will have no way to accurately reconcile discrepancies in the data.
Last edited:
And again, Apple sells Apple Watch for consumers.
I’m sure Apple LOVES to boast about researchers using Apple Watch for medical researches, and in that sense they will find a way to work with them somehow. The popularity of using Apple Watch for researches may catch Apple off guard who knows. I’m just saying it’s good that Apple still try to improve their algorithm whenever they can since it’s one of the most important thing in the watch.
You don’t know that. Please stop knee-jerking.
If Apple involves they could provide all the data researchers need.
Apple is not caught off guard on this. They worked with WHO on several studies using the Watch as the tool for taking measurements.
True they did. I didn’t see those complaining. If you have knowledge that I don’t know of please share a link.
I worked for 30 years doing computer simulation of petroleum reservoirs. Figuring out the history to match prior to prediction mode was half the work.
Yeah, the VO2 max calculations are massively off too, and they’ve fiddled around with that algorithm regularly too.
Apple’s calculation is consistently less than two thirds of my actual, and they claim this has been validated, not that they provide any evidence.
I filed a feedback about it around a year ago with detailed information as to how I discovered it was so inaccurate - I’ve had no response.
I have read a lot of scientific research papers about the value of resting heart rate and hrv, and they all come to the same conclusion: hrv measurements taken at any time other than in the morning right after you wake up, have absolutely no scientific value whatsoever. I don’t know what other peoples needs you are referring too, but i doubt it’s valuable.
btw, i feel backed up by this article as well.
The FDA has not certified that feature to a high level so it falls under the fun and wellness category. One level above toy. Too many people are giving the watch unwarranted praise and attention and Apple their cash.