Question Re Calibration/Running Discrepancy

[viewfly]

Opinion question for you guys. I've read the watch manual, and the Apple tech article on calibration several times. I think I understand better, but with one uncertainty.

First, the Watch is mainly a calories tracker for read out.

Second, when in 'Outdoors ' exercise mode, and with the iPhone on you, it ALWAYS uses GPS for distance measurements. Regardless if in calibration mode or not.

Third, indoor mode, or outdoor mode, with no phone or poor GPS, it will use stride for distance, which is based on height, gender, etc, like most simple devices. Unless you turn on the calibration mode to force stride calibration, it will then made many calibrations based on your speed, so cover as many strides as needed.

Indoors, calories is based mainly on heartbeat, outdoors, it is mainly based on distance and time. Straight from the manual, see below.

So the question, is this: the manual also says, outdoor, it is continually updating your stride calibration, WITHOUT, stating you must be in calibration mode! What does this mean? See 2nd excerpt below.

My guess, and only a guess: in calibration mode, it forces stride calibration, assuming you have good GPS, per the article. With calibration mode off, it may or may not use GPS, based on the quality or existence of a GPS signal. It depend if the algorithm is good. So, perhaps, doing a calibration in a poor or interrupted GPS, will ruin your calibration somewhat. Maybe the best bet is do the calibration correctly in ideal contains, and then turn it off.

What do you think? It not easy getting a precise answer in this.

Regarding pace, it is BS on the watch unless you have a consistent stride outside. It seems mainly to want to figure calories in this OS. So run for 30 minutes, it measures time and distance with GPS; if you pause for minutes, it just figures total time/distance for calories ( with some heart rate data) during that 30 min, and the average pace is just what you did over that 30 min, regardless if you stopped...which is fair, understanding what it is doing.

I think the watch is doing a great job, but the Activity readout is geared for calories usage, rather than for actual pace, like an app such as Runkeeper or Runmeter, etc. Perhaps Apple and/or 3 rd parties will pull in the data in a better form in the future for serious runners.


“Note: Outdoor and Indoor Walk/Run/Cycle are distinct workouts because Apple Watch calculates the calorie burn differently for each. For indoor workouts, Apple Watch relies mainly on your heart rate readings for calorie estimates, but for outdoor workouts, Apple Watch works in conjunction with iPhone (which has GPS) to calculate speed and distance. Those values, along with your heart rate, are used to estimate the number of calories burned.”

Excerpt From: Apple Inc. “Apple Watch User Guide.” Apple Inc., 2015. iBooks.
This material may be protected by copyright.

Check out this book on the iBooks Store: https://itun.es/us/4HEW6.lp

“Your iPhone GPS allows Apple Watch to achieve even more distance accuracy. For example, if you carry iPhone while using the Workout app on a run, Apple Watch uses the iPhone GPS to calibrate your stride. Then later, if you’re not carrying iPhone, or if you’re working out where GPS is unavailable (for example, indoors), Apple Watch uses the stored information about your stride to measure distance.”

Excerpt From: Apple Inc. “Apple Watch User Guide.” Apple Inc., 2015. iBooks.
This material may be protected by copyright.

Check out this book on the iBooks Store: https://itun.es/us/4HEW6.l

 

[exxxviii]

Great analysis and digging. Here are my guesses between the gaps...

At a high level, the watch uses the same simple steps * stride length approach to get to distance that every other fitness tracker uses. It starts with a default walking and running stride length based on height and sex. Then, it will override those defaults with data it calculates from calibration.

The watch does not have a formal calibration mode that we can control. Rather, it will dynamically calibrate any time that it is connected to the phone during an outdoor workout. My guess is that it still uses stride and steps to get to distance with an outdoor workout, even while calibrating. Apple recommends 20 minutes of workout activity to acquire sufficient calibration data (https://support.apple.com/en-us/HT204516). This implies that the watch has a database that contains at least 20 minutes worth of calibration data. Which further suggests that during dynamic calibration the watch must be deciding which of the older calibration data to replace with new data.

I do not think that the watch is using GPS for distance measurement except for Outdoor Cycle. The manual is not clear, but I believe that it is only using GPS to calibrate stride length for Outdoor Run and Outdoor Walk, and then using that stride length to reach a distance. GPS is involved, but it is secondary rather than primary.

Your point about calibrating with dodgy GPS is spot on. The problem is that my experience with iPhone GPS is that it will just happen, and you never know when. So, maybe that is why Apple wants 20 minutes of calibration data-- so that the errant bad GPS readings will be diluted by many more good readings.

As long as you have good stride data in the watch, then there is no reason it cannot be decent on pace. It will be as precise as a GPS watch, but it will be OK (5% or better) under most circumstances. If you are on a run and shift between jogging or walking, then pace and distance will be way off. But otherwise, it can do a decent job.

My rub with it is that it is not doing as good a job as my Fitbit where I manually enter the stride lengths for walk and run. That means that Apple's calibration algorithms suck.

 

[viewfly]

I do not think that the watch is using GPS for distance measurement except for Outdoor Cycle. The manual is not clear, but I believe that it is only using GPS to calibrate stride length for Outdoor Run and Outdoor Walk, and then using that stride length to reach a distance. GPS is involved, but it is secondary rather than primary.

Thanks for replying, and I pretty much agree, but some points are not so clear.

The manual clearly states that GPS is used in outdoor mode for distance, and not stride ( from the accelerometer). This makes sense since 1) the calibration mode ( have Motion & Distance calibration ON) is not the default on new watch set up, and 2) even when walking for 30 min with a full stop for 4 minutes, the distance is only 0.01 mile off from a Runmeter app. It can't be using stride.

And I think it is using multiple hours of data tables to refine your various stride calibration from GPS data and the accelerometer in the watch. And it gets better with time. I think that is better than throwing away data.

My main question is do we keep the calibration setting on or off, after doing the cal procedure, and why? What's the downside of keeping it on? Because, since it is off by default and yet Apple says the Watch still uses GPS to improve the stride for when GPS is unavailable and caloric measurements.

Mystery.

 

[exxxviii]

My main question is do we keep the calibration setting on or off, after doing the cal procedure, and why? What's the downside of keeping it on? Because, since it is off by default and yet Apple says the Watch still uses GPS to improve the stride for when GPS is unavailable and caloric measurements.

I interpreted the manual's statement "if you carry iPhone while using the Workout app on a run, Apple Watch uses the iPhone GPS to calibrate your stride" to mean that the phone's GPS only affects the stride length factors in the watch; that it does not override directly override the distance the watch displays.

I do not think there is a way to stop the watch from dynamically calibrating without diminishing the function of the phone as a whole.

 

[viewfly]

I interpreted the manual's statement "if you carry iPhone while using the Workout app on a run, Apple Watch uses the iPhone GPS to calibrate your stride" to mean that the phone's GPS only affects the stride length factors in the watch; that it does not override directly override the distance the watch displays.

I do not think there is a way to stop the watch from dynamically calibrating without diminishing the function of the phone as a whole.

We certainly need someone deep inside Apple to clarify this, if they even wish to.

From what I read, A-GPS maybe has at best 8 meter resolution (24 foot diameter ) so I don't think it can calibrate your tiny stride in real time, step by step accurately. It needs to use the GPS over a longer distance, hence the 20 minute rule ( a mile for a walker, more for a runner).

So in my mind, the Apple statement means it measures distance via GPS, and measures your stride by the accelerometer, and then calculates an average step/ distance for each of your pace speeds.

Perhaps the difference between doing this with the cal setting on/off is the sampling rate of using the GPS. Off would be coarser and save battery life on the iPhone. But once calibrated in stride, it will still give a good result.

???

Certainly all the measurements and sensors are there. I see no way the Garmin GPS is better than the iPhone GPS. Nor any accelerometer.

Perhaps it is important to let the GPS fully lock onto many satellites at the run beginning for best accurately.

More to think about.

 

[exxxviii]

Perhaps the difference between doing this with the cal setting on/off is the sampling rate of using the GPS. Off would be coarser and save battery life on the iPhone. But once calibrated in stride, it will still give a good result.
...
I see no way the Garmin GPS is better than the iPhone GPS. Nor any accelerometer.

There is not an on/off calibration setting for the Watch.

Motion Calibration on/off on the phone calibrates the M7 or M8 motion co-processor. Location services on the phone enables the use of GPS. You can cripple the phone by turning both off if you want to prevent the watch from calibrating; otherwise the watch will dynamically calibrate whenever it is connected to the phone during a workout.

I have thousands of miles of runs logged with Garmin GPS watches and hundreds of miles with RunKeeper on my iPhones. The Garmin is accurate to around 0.2% in distance (standard deviation less than 0.01 miles over a 5 mile run). The phone with RunKeeper is about 10 to 15 times worse than the Garmin watch. It could be the app, but other apps have appeared to be about the same as RunKeeper. In my experience, I have observed that Garmin GPS is superior to iPhone GPS by one order of magnitude.

 

[NotSafeForWork]

Don't even try on the treadmill indoors. It can be as much as 30% off on distance (Apple Watch always over-optimistic).

I have tried outdoor calibration and it makes no difference. Apple really need to provide a calibration feature - run a mile on the treadmill then "edit" the Apple Watch distance back from 1.3 to 1.0 miles for "Indoor Running".

 

[exxxviii]

Don't even try on the treadmill indoors. It can be as much as 30% off on distance (Apple Watch always over-optimistic).

I have tried outdoor calibration and it makes no difference. Apple really need to provide a calibration feature - run a mile on the treadmill then "edit" the Apple Watch distance back from 1.3 to 1.0 miles for "Indoor Running".

Yeah, this is a fail. I don't think that the Apple designers know that runners run differently, with a quicker and more efficient stride, on a treadmill. So, if you run with a cadence that is 5% faster on a treadmill with a shorter stride length, then the watch's guess at distance will be way off.

 

[viewfly]

There is not an on/off calibration setting for the Watch.

Motion Calibration on/off on the phone calibrates the M7 or M8 motion co-processor. Location services on the phone enables the use of GPS. You can cripple the phone by turning both off if you want to prevent the watch from calibrating; otherwise the watch will dynamically calibrate whenever it is connected to the phone during a workout.

I have thousands of miles of runs logged with Garmin GPS watches and hundreds of miles with RunKeeper on my iPhones. The Garmin is accurate to around 0.2% in distance (standard deviation less than 0.01 miles over a 5 mile run). The phone with RunKeeper is about 10 to 15 times worse than the Garmin watch. It could be the app, but other apps have appeared to be about the same as RunKeeper. In my experience, I have observed that Garmin GPS is superior to iPhone GPS by one order of magnitude.

You made a good point. Maybe that is the center of the problem. It made me remember how cell GPS can work.

We write about the smartphone 'GPS'. And compare it to a Garmin GPS, which is stand alone. A smartphone uses A-GPS, combinations of cell towers, and maybe wifi, if any are around. Regardless they degrade the location resolution, from what I've read, compared to plain GPS. Even with plain GPS, resolution depend on # of satellites in view, time to acquire etc.

If I remember GPS correctly, 0.01 mile, or 52 feet is about right, and is what you're going to get for 5 miles, 1 mile, or 500 miles. But a cell tower, 'helping', with A-GPS, has poor resolution, maybe 8x.

Hikers force their smartphones to use only the GPS, to save battery, by turning ON the airplane mode, turning off the cellular data. It works. Turn on airplane mode, and then turn back on Bluetooth to communicate with the watch. I tried it and it works, and I can even run RunMeter from the watch. And the watch runs too.

Maybe someone wants to try calibration ( System Services ON for Motion and Distance, the on/ off calibration procedure according to Apple ) in Airplane mode and walk or run that 20 minutes. Curious what it does, if the accuracy improves.

Just a thought.

 

[Snoopy16]

Just wanted to say thanks for all the great info and thoughts!

 

[iPhysicist]

I did some experimenting with my watch.

Unfortunately I had to un-pair my spot on calibrated watch due to iOS9 beta issues. As a result I lost my calibration and then could not justify to go to the nearby river (flat area without high buildings) to do the calibration again. So I did a run without calibrating.
The watch knows my length and can therefore calculate a (more or less accurate) stride. What it can not do is assume a speed. It needs data to do it. My standart lap is 2km (I took the data from google) +/- 50m (for not running the optimal path). The first km mark was of by + 200m. The second km was still of but a little less. Then the third km was about +/-25m where I was at the first km. The fourth km mark and the sixth km mark where exactly at the same spot.

I did this test twice with different pace (5:30+' /km and 4+'/km) with the same results. The not calibrated watch gets more accurate with every meter you run. This of course means you have to run at a constant pace. I tried to do so but as often you get faster after the first lap.

The problems with accuracy and side by side comparison is:

1. The people complaining about accuracy of the watch (only) mostly run without a proper calibrated watch.
2. GPS watches do have measurement errors too.
3. Running at different speeds (or uphill, downhill)

 

[viewfly]

Nice experiment and analysis! Thanks

 

[viewfly]

Motion and Distance Calibration off. Walked 3 miles with iPhone in pocket. Can see the arrow icon on the phone confirming that GPS is being used in 'Outdoor' mode, even if not doing calibration.

The iPhone is using the watch in conjunction with the iPhone GPS to measure your distance.

 

[exxxviii]

Motion and Distance Calibration off. Walked 3 miles with iPhone in pocket. Can see the arrow icon on the phone confirming that GPS is being used in 'Outdoor' mode, even if not doing calibration.

The iPhone is using the watch in conjunction with the iPhone GPS to measure your distance.

I think what you are seeing is the watch using the phone's GPS to calibrate stride length (as the manual describes). The phone's Motion and Distance Calibration setting applies to the phone's M7/M8 motion co-processor and is independent of the watch. Remember that the iPhone 5 does not have the motion co-processor chip, so you effectively made the phone & watch combo the equivalent as if you had an iPhone 5.

 

[viewfly]

I think what you are seeing is the watch using the phone's GPS to calibrate stride length (as the manual describes). The phone's Motion and Distance Calibration setting applies to the phone's M7/M8 motion co-processor and is independent of the watch. Remember that the iPhone 5 does not have the motion co-processor chip, so you effectively made the phone & watch combo the equivalent as if you had an iPhone 5.

I'm not sure what you're saying here. But I think we are saying the same thing, but just in a different way.

Any GPS's resolution is too crude to measure a persons single stride ( 2-3 feet). Instead it has to be that the iPhone GPS measures some larger distance, and the watches's accelerometer measures the number of swings of your arm. Dividing the total distance/# arm swings calculates your stride. The longer the distance ( in a good GPS area), means more arm swings, and that gives a more precise stride calibration, on average. Apple does this to make "it just works", since the average person never bothers to measure their stride and input it. Normally just height and gender is used, and that is not so accurate.

All this is assuming your arm swing relates to your leg stride, which is not too far off. I suppose some ' bounce' sensing may be in the measurement too. IDK.

The GPS measures a distance, but is too imprecise to measure a persons single leg stride distance. Hence the 20minute requirement, which gives good precision. Stride is calculated with the GPS, not measured. This is what I mean.

I may have been mistaken when I saw the GPS icon on during an Outdoor Walk. I can't replicate it. But it is on when I set 'Motion & Distance' ON and an outdoor walk.

Some interesting links showing the accuracy of the Apple Watch against other smart watches and FitBits, and GPS watches. Seems spot on with these reports.

http://www.cnet.com/news/smartwatch-step-counter-and-distance-tracker-accuracy/

http://www.idownloadblog.com/2015/04/26/early-apple-watch-tests-accurate-health-fitness-tracking/

 

[exxxviii]

But I think we are saying the same thing, but just in a different way.

Yep, we are saying the same thing. When I read "distance," I thought you meant that the distance captured by the phone was going directly to the distance shown on the watch for the workout. I agree that the watch is capturing a longer distance and then dividing steps to get stride.

Those two articles are interesting. The CNET dude should get flogged-- he compared the Apple to effectively un-calibrated devices. Talk about apples and oranges. Although he acknowledged that the other devices allow users to manually calibrate, he decided that "most people don't bother to calibrate those activity trackers and smartwatches that do include the feature." Therefore, it appears that he left them uncalibrated for his comparison. My manually calibrated Fitbit Zip is more accurate than the calibrated AW.

I would love to know what kind of black magic Hannes Verlinde used to get such similar results between his Garmin 610 and the AW. The Garmin 600-series is one of the best running watch lines on the planet, so its distance is probably pretty trustworthy. Maybe 1 run out of 20 has been that close for me, with the AW consistently 2% shorter distance than the Garmin. My AW gets points for its consistency-- but it is consistently wrong. I blame the calibration. I suspect that if the AW is lucky enough to calibrate accurately, then the results will be accurate. But, accurate calibration may be a crapshoot.

 

[viewfly]

Yep, we are saying the same thing. When I read "distance," I thought you meant that the distance captured by the phone was going directly to the distance shown on the watch for the workout. I agree that the watch is capturing a longer distance and then dividing steps to get stride.

Those two articles are interesting. The CNET dude should get flogged-- he compared the Apple to effectively un-calibrated devices. Talk about apples and oranges. Although he acknowledged that the other devices allow users to manually calibrate, he decided that "most people don't bother to calibrate those activity trackers and smartwatches that do include the feature." Therefore, it appears that he left them uncalibrated for his comparison. My manually calibrated Fitbit Zip is more accurate than the calibrated AW.

I would love to know what kind of black magic Hannes Verlinde used to get such similar results between his Garmin 610 and the AW. The Garmin 600-series is one of the best running watch lines on the planet, so its distance is probably pretty trustworthy. Maybe 1 run out of 20 has been that close for me, with the AW consistently 2% shorter distance than the Garmin. My AW gets points for its consistency-- but it is consistently wrong. I blame the calibration. I suspect that if the AW is lucky enough to calibrate accurately, then the results will be accurate. But, accurate calibration may be a crapshoot.

In the CNET table, both uncalibrated and calibrated AW results are given. The results very close but I agree he should have calibrated the other devices. In practice the Apple calibration method should be the most accurate

The second article confirms my experience. The AW distance is spot on for me. Pace only is accurate if I don't pause and maintain an even gait

I still don't understand Apple's manual. It says GPS is used in calibration mode, but also in a workout when the Cal & Distance switch is off. ??? I can confirm the former but not the latter.

 

[exxxviii]

Well, I just got off the phone with Apple support and they confirmed that an un-pair/re-pair is the only way to calibration data. You lose Apple Pay plus minor miscellaneous things. It is kind of ridiculous, actually. Recalibrate should just be a button on either the Workout app or on the iPhone watch app. I am probably not going to mess with it.

Apple fixed this. They gave us a "Reset Calibration Data" link in the Motion & Fitness section of the Privacy Settings in the AW phone app. Hooray.

 

[viewfly]

Great! Now you have lots of options. Is the accuracy better for you now?

 

[exxxviii]

Great! Now you have lots of options. Is the accuracy better for you now?

No, in fact it is far worse since the OS 2.x updates. I am planning to clear the watch calibration again to see if it gets any better. The silly watch reports 5.6 to 5.7 miles on a 5.07 mile run. And this is after hours of time wearing the watch and phone together.

 

[aarontsuru]

I run the same set up. Keep in mind, iSmoothRun has autopause where Apple Workout does not. While it gets the miles the same within reason, iSmoothRun is significantly more accurate on time actually running and pace and all that since it's not counting when I'm stuck at a light in Manhattan or getting a sip of water at the park or stretching the moment before I run or just catching my breath for a second on a hill.

I use the Apple Workout app strictly for circles, heart rate, and calories. iSmoothRun's data for actual run analysis. Then, when I export to strava I add the Apple Workout heart rate & calorie info to the export.

For example, here's today! https://www.strava.com/activities/428895318

 

[viewfly]

No, in fact it is far worse since the OS 2.x updates. I am planning to clear the watch calibration again to see if it gets any better. The silly watch reports 5.6 to 5.7 miles on a 5.07 mile run. And this is after hours of time wearing the watch and phone together.

Huh. My watch is within 0.05 miles or less of any device I cross check it with.

 

[exxxviii]

Huh. My watch is within 0.05 miles or less of any device I cross check it with.

I did my first re-calibration run this morning after resetting the calibration data. The watch measured 5.10 miles on a 5.07 route. I will give it another 20 miles with the phone and then do a few runs without to see how it does.

 

[BlueMoon63]

I did my first re-calibration run this morning after resetting the calibration data. The watch measured 5.10 miles on a 5.07 route. I will give it another 20 miles with the phone and then do a few runs without to see how it does.

I saw a good article on cult of Mac and how poorly the Apple Watch performed without the iPhone along for the ride. I calibrated a bunch of times but run or walk the same route so the watch by itself has been accurate. When I do intervals with more pronounced arm movements it seems to start to be off. I normally barely move my arms when I run and it seems accurate.

I would never trust the Apple Watch for serious running though.

 

[exxxviii]

When I do intervals with more pronounced arm movements it seems to start to be off.

Your interval issue is probably stride length, not arm movement. When you are running fast, you are extending your stride length significantly. This will cause the AW to think you went a shorter distance than you actually ran, because it has a shorter stride length in its calibration data from your normal stride.

The frustrating thing that I am still seeing is that my statically calibrated Fitbit seems more accurate than the AW's dynamic calibration.