Since I saw the tests that Consumer Reports did, I wanted to do my own tests just to be sure. So far I have been testing it against my Polar V800 (a high end GPS fitness watch). I intend to do more testing to try to reproduce any inaccuracies that you claim.
Very cool. I look forward to your follow-up. Here are a few quick thoughts to help jump start your analysis. First, if you had your watch with OS1, you probably know that calibration was different then. Many folks thought it was much more accurate. That changed with OS2. I have a 5.07 mile route that I run regularly. You will want to establish a validated distance to test against. I recommend a few different GPS watches, a favorite running app on the phone, and surprisingly, an un-calibrated AW running with the phone. When I run this route, my Garmin will measure the distance within a 0.01 mile standard deviation. Runkeeper on the phone will be within about 0.1 mile SD. The AW without calibration and running with the phone will also be within 0.1 miles SD. After calibration, my AW measures 5.65 miles with .15 mile SD. I recommend you capture dozens of readings for it to be statistically valid.
Also, you will want to measure running and walking separately. I do not walk with my AW, but I have read that walking distances and calibration are much more accurate with the AW. It is running where people have observed the most problems.
Next, for calories, you will want to compare the watch's calorie counts to some other devices and the various online running calorie calculators to get some data. If you start with outdoor running, you will see decent consistency between the AW and other calculators and devices. But, the AW starts to vary as you get to indoor running and cycling. I am a multi-sport athlete, so I have a pretty good judge of my body's level of effort across activities. My Garmin and a high-end treadmill calculate calories pretty closely, but the AW was off by a couple hundred. I got similar readings on indoor cycling-- with the AW about half of the Garmin. I totally trust the Garmin's calculation.
Finally, if you had an AW with OS1, you saw that the AW base metabolic rate for calorie burn was off. Apple fixed it with OS2, but there are still some issues. If you look at your workouts, and subtract total calories from active calories, you get the BMR calories burned during the workout. Then, factor that to calories per hour. In my case, the AW is consistently 114 calories per hour during my workouts, which translates to 2,736 BMR calories per day. When I had OS1 on the watch, it thought my BMR was on the order of 2,800 calories per day. So, my hunch is that Apple only partially fixed the calorie calculation in OS2, but the errors still show up within workouts.
EDIT: I thought of one other thing... you will want to gather running distance data with and without the phone present. The AW gives different distance readings depending on whether it was connected to the phone during the workout. For example, on a whim I ran with my phone today, and the AW distance was 0.16 miles shorter than the exact same run last week without the phone.
