Stress Testing the iPhone

[MacRumors]

The USA Today article we had previously referenced also contains an interesting sidebar interviewing Balsu Thandu, one of the individuals who has been field testing the iPhone over the past two months.

Approximately 200 field technicians have been secretly testing the iPhone for 10 weeks, logging over 10,000 hours on the phone, including over 5,000 hours of voice calls and 5 gigabytes of data usage. This is reportedly twice as much testing as other phones.

Users worried about durability can rest easier that testing included dousing the iPhone in water, dropping it onto concrete and bouncing it off sidewalks. Real life usage testing was also a big focus:

"Many people don't realize it, but walking gives you the worst channel conditions," Thandu says. Cell signals tend to bounce off buildings, causing interference, and background noise is a constant problem in cities.

Apple reportedly made ongoing adjustments based on the feedback, including improving audio quality which was originally "not loud or clear enough".

According to the article, Thandu feels comfortable that the iPhone is ready for launch.

[Grimace]

awesome!! way to test it beforehand!

That sounds pretty thorough -- good to see that Apple knows that the spotlight is shining pretty brightly. This needs to be a flawless launch.

[Ja Di ksw]

Glad to hear it, helps me feel it's more durable. I'll still wait for the 3rd revision, but it makes me feel better.

[Eraserhead]

Originally Posted by Macrumors

Users worried about durability can rest easier that testing included dousing the iPhone in water, dropping it onto concrete and bouncing it off sidewalks.

Sounds like an Arstechnica review.

[xUKHCx]

Wonder how it will stand up against the torrent of abuse that it will receive in my pocket complete with my keys and coins.

[Dr.Gargoyle]

Originally Posted by xUKHCx

Wonder how it will stand up against the torrent of abuse that it will receive in my pocket complete with my keys and coins.

The real test will be when thousands of users begin to abuse it in 6 days. I guess we will know for sure within a month or so

[Zadillo]

That actually doesn't seem like all that much, given the nature of the phone (meant to browse full websites, not just WAP-formatted sites).

Considering that Verizon limits their "unlimited data plan" to 5 gigabytes a month, I would have sort of expected more from 200 field testers.

I know its EDGE and not 3G, but even so (I'm assuming they only meant EDGE data usage and not wifi data usage)........ given how much testing would have presumably been done with full webpages (which can easily be a megabyte per page of data), 5 gigabytes just doesn't seem like much.

[xUKHCx]

Originally Posted by Dr.Gargoyle

The real test will be when thousands of users begin to abuse it in 4 days. I guess we will know for sure within a month or so

The only reason why i'm glad that it is not getting released in the UK for a couple more months. Will be refined over that period and we will end up with a better iPhone on release.

[Much Ado]

The more i read, the more it seems Apple are going at this at full throttle.

Whether you like the iPhone or not, you've got to say- the job they've done so far is pretty immaculate.

6 days...

[thejadedmonkey]

Originally Posted by Macrumors

According to the article, Thandu feels comfortable that the iPhone is ready for launch.

Awesome, I'm glad that a phone 6 days away from launch is ready for launch

[Dr.Gargoyle]

Originally Posted by xUKHCx

The only reason why i'm glad that it is not getting released in the UK for a couple more months. Will be refined over that period and we will end up with a better iPhone on release.

Agreed. I think I will stay away from the Rev. A. and take the plunge when Rev. B. hits the market. It will hopefully sport onboard GPS and 3G (perhaps even HSDPA).

[Zadillo]

Originally Posted by thejadedmonkey

Awesome, I'm glad that a phone 6 days away from launch is ready for launch

Considering how many companies launch hardware and software before its ready, this isn't something I'd take for granted.

More specifically, it's sort of remarkable how many phones seem to get churned out without even the most rudimentary user testing..... only after the phone comes out and people discover obvious problems (keypad is uncomfortable, camera doesn't work, etc.).

[crazzyeddie]

Originally Posted by Zadillo

That actually doesn't seem like all that much, given the nature of the phone (meant to browse full websites, not just WAP-formatted sites).

Once you take Flash out of the equation then that is alot of data. 5GB is enough to get a good estimate of what to expect anyway. 5GB of transfer would be about equivalent to what a regular person would do in a week or so of normal browsing.

Note: Most MR/Digg users don't qualify as normal.

[SheriffParker]

I like how Apple fixed the audio quality. At least some changes were made beause of the testing.

[kcroy]

Sounds great. I will buy this first model. I'l probably kick myself when the next generation arrives and my iPhone is missing a feature, but my Alltel contract is up and my current phone is absolute horrid junk! Can't wait until I have my iPhone. Getting one for the wife too.

[Zadillo]

Originally Posted by crazzyeddie

Once you take Flash out of the equation then that is alot of data. 5GB is enough to get a good estimate of what to expect anyway. 5GB of transfer would be about equivalent to what a regular person would do in a week or so of normal browsing.

Note: Most MR/Digg users don't qualify as normal.

Right, but from the way I read it, I thought they meant 5GB was the total amount of bandwidth used among all the testers. That is, all 200 testers logged a total of "over 10,000 hours on the phone, including over 5,000 hours of voice calls and 5 gigabytes of data usage"

5GB sounds very good for a single user for an Internet-centric phone, but if it's for all 200 testers, it seems like not a whole lot.

-Zadillo

[synth3tik]

Thats nice to know considering that my phone broke 1 week broke the iPhone launch.

[wnurse]

Originally Posted by Much Ado

The more i read, the more it seems Apple are going at this at full throttle.

Whether you like the iPhone or not, you've got to say- the job they've done so far is pretty immaculate.

6 days...

Every phone manufacturer test their phones this way. This is nothing unique. Still says nothing about real world usage (if that was the case then there would never be a bad phone since they are all tested)... does say though that apple is doing their homework. At least they are not hoisting an untested phone on the public.

[gotohamish]

Originally Posted by Much Ado

The more i read, the more it seems Apple are going at this at full throttle.

Whether you like the iPhone or not, you've got to say- the job they've done so far is pretty immaculate.

6 days...

Indeed, I hope they do this more for their laptops too.

[123]

Originally Posted by article

Apple reportedly made ongoing adjustments based on the feedback, including improving audio quality which was originally "not loud or clear enough".

Hmm, I don't like this. In 10 weeks you can improve audio quality, but you can't make it dramatically better or excellent.

[rtdunham]

Originally Posted by Dr.Gargoyle

Agreed. I think I will stay away from the Rev. A. and take the plunge when Rev. B. hits the market. It will hopefully sport onboard GPS and 3G (perhaps even HSDPA).

I don't want to wait. Is there a downside to buying rev A, knowing i'll replace it if a better Rev B appears?

--I've always been able to sell prior-gen iPods online, is there anything that will keep the iPhone from being an attractive sale item? i mean, isn't it likely someone who buys one could take it to a store for activation?

--What are ATTs usual policies for upgrading phones? Do you just extend the contract 24 mos from the new purchase? they wouldn't charge you a cancellation fee if you're doing that, would they?

[ajhill]

Originally Posted by synth3tik

Thats nice to know considering that my phone broke 1 week broke the iPhone launch.

Somebody is trying to tell you something...

[Zadillo]

Originally Posted by rtdunham

I don't want to wait. Is there a downside to buying rev A, knowing i'll replace it if a better Rev B appears?

--I've always been able to sell prior-gen iPods online, is there anything that will keep the iPhone from being an attractive sale item? i mean, isn't it likely someone who buys one could take it to a store for activation?

--What are ATTs usual policies for upgrading phones? Do you just extend the contract 24 mos from the new purchase? they wouldn't charge you a cancellation fee if you're doing that, would they?

It depends, normally you have to be pretty far into your contract to get upgrade pricing on a new phone.

But as for the iPhone, it wouldn't seem to matter, since there aren't different "upgrade/new customer" prices.

I'm actually not even clear on what a current Cingular customer would do who wasn't in an upgrade period. I think I've read that they would just extend your contract.

I have to imagine that if you want to buy the next revision iPhone as well when it comes out there will be a way to do it.

[Zadillo]

Originally Posted by 123

Hmm, I don't like this. In 10 weeks you can improve audio quality, but you can't make it dramatically better or excellent.

How can you possibly know what they could or couldn't do?

All that matters I think would be that the audio quality on the final product is good, and so for there aren't any reports to the contrary.

[strangelogic]

Originally Posted by wnurse

Every phone manufacturer test their phones this way. This is nothing unique. Still says nothing about real world usage (if that was the case then there would never be a bad phone since they are all tested)... does say though that apple is doing their homework. At least they are not hoisting an untested phone on the public.

This was at&t testing team, not if I read it correctly. And while all companies do testing on new products, all testing is not the same. I've worked in s/w testing specifically for 20 years - and despite a documented methodology and extensive training there are systems that crash/burn at launch due to poorly planned/executed tests and incompetent testers.

The implication of the article to me is that this at&t testing was more than they usually do because they normally just rely more on the manufacturer testing. It does not mean it will be perfect by any means. But, it does mean that they have been making every effort to make sure it doesn't suck.