iOS4 and iPhone 4 Wreaking Havok in the corporation

T

ToxicMan

macrumors member
We have a very large mobile population in my company with thousands of apple iPhones and a growing population of iPads. We support a virtual desktop and mobile application suite.

Starting Monday, our users began upgrading to iOS4 and on Wednesday and Thursday; many more started receiving iPhone 4.

Beginning Tuesday, our helpdesk began taking increasing numbers of calls reporting intermittent loss of mail and calendar synchronization with our Microsoft Exchange services.

By Friday, almost all activsync services are down affecting all our devices with iOS3 including the entire iPad population.

There are some threads over on the apple forums suggesting a DNS change from "CNAME" to "A" type record for activesync (generically referred to in the threads as Exchange). However this has not turned out to be a solution to the issue.

At this point, we have suspended adoption and are actively working to find a way back.

Will keep this forum posted...
 
ToxicMan said:
We have a very large mobile population in my company with thousands of apple iPhones and a growing population of iPads. We support a virtual desktop and mobile application suite.

Starting Monday, our users began upgrading to iOS4 and on Wednesday and Thursday; many more started receiving iPhone 4.

Beginning Tuesday, our helpdesk began taking increasing numbers of calls reporting intermittent loss of mail and calendar synchronization with our Microsoft Exchange services.

By Friday, almost all activsync services are down affecting all our devices with iOS3 including the entire iPad population.

There are some threads over on the apple forums suggesting a DNS change from "CNAME" to "A" type record for activesync (generically referred to in the threads as Exchange). However this has not turned out to be a solution to the issue.

At this point, we have suspended adoption and are actively working to find a way back.

Will keep this forum posted...
Click to expand...

I suppose its possible. Tell them to fix their settings.
 
It just so happens after every iPhone release many new people join our forums, (hence macrumors newbie title) and if a few people have a slight issue, they stretch it like crazy and troll the hell outta it.
 
klamse25 said:
It just so happens after every iPhone release many new people join our forums, (hence macrumors newbie title) and if a few people have a slight issue, they stretch it like crazy and troll the hell outta it.
Click to expand...

Thanks for the support, sort of. I typically don't post on fanboy sites, but because this is impacting so many of my users and nothing is showing up yet on general Google searches, I'm looking here and at apple forums.

I suppose this is the wrong place given it's fanboy consumer user base and thus a "newbie" to the board is welcomed with open arms, NOT.

It's this exact reason I tend to stay away from consumer forums and only contribute to professional association related user communities (like ASUG)

But given the impact this is having within my shop, I'm looking everywhere for ideas (I have an open P1 case with Microsoft PS as well). Oh, and for those that don't know what Microsoft PS is, it's Microsoft Premier Support.

Don't mean to be pedantic, but I'm in no mood right now for contributions that aren't helping.

Please don't post if you have nothing worth contributing. I'm happy to chase down valid technical solution leads!!!
 
How much testing did you do?

In an environment like that, you should have been testing all the way through gold brick release. If you did, why didnt this problem rear its ugly head then?

Just curious.
 
xxSpudxx said:
How much testing did you do?

In an environment like that, you should have been testing all the way through gold brick release. If you did, why didnt this problem rear its ugly head then?

Just curious.
Click to expand...

Couldn't agree more. Not happy with some folks who didn't get on the developers build and now I'm left to clean up.

We just "reorganized" that group today... read between the lines.

ARGH!!!
 
UPDATE:
I was working with Apple over the weekend and have a patch for the iOS4 devices that extended the timeout settings on synchronization. We also added additional ActiveSync Servers which has solved the problem even for those that do not have the patch. It looks like the issue is still under investigation but from what we can tell so far, iOS4 made some changes to the way it interacts with Activesync and increased the traffic substantially. As a result, our infrastructure was overwhelmed by the additional traffic.

Those that posted on the Apple forums about reducing the size of the mailbox would have a similarly successful outcome because they're effectively reducting the load on the server. However, we opted to increase capacity by scalling out our activesync infrastructure.

We have also removed the patch from Apple on many of the affected devices and those devices continue to work well again without any additional issues.

Therefore, we have a temporary solution and will continue to work with applecare to determine the root cause in iOS4.
 
We have no issues that I am aware of w/ iPhone 4s and iOS 4 upgrades at my work.


(I'm the IT guy so I would know about it)
 
ToxicMan said:
We have a very large mobile population in my company with thousands of apple iPhones and a growing population of iPads. We support a virtual desktop and mobile application suite.

Starting Monday, our users began upgrading to iOS4 and on Wednesday and Thursday; many more started receiving iPhone 4.

Beginning Tuesday, our helpdesk began taking increasing numbers of calls reporting intermittent loss of mail and calendar synchronization with our Microsoft Exchange services.

By Friday, almost all activsync services are down affecting all our devices with iOS3 including the entire iPad population.

There are some threads over on the apple forums suggesting a DNS change from "CNAME" to "A" type record for activesync (generically referred to in the threads as Exchange). However this has not turned out to be a solution to the issue.

At this point, we have suspended adoption and are actively working to find a way back.

Will keep this forum posted...
Click to expand...

Does your IT deparment not have an approved list of devices? Cleary they should have tested prior to allowing them. ISsues with active sync are not limited to the iPhone and are quite frequent with new devices that is why we test and approve.
 
pugnut said:
Does your IT deparment not have an approved list of devices? Cleary they should have tested prior to allowing them. ISsues with active sync are not limited to the iPhone and are quite frequent with new devices that is why we test and approve.
Click to expand...

Asked and Answered.
Expanded...
iPhone is approved device.
There is no way to STOP users from upgrading to iOS4, as the option is presented to all users when they dock!

Could not predict the impact because we could not tell in advance that iOS4 handles activesync differently than prior versions which created rapid and significantly more traffic to ActiveSync services, which appeared as a result of the increased use of the new iOS.

We're up and working, and continue to work with Applecare and Engineering directly to see whats caused the dramatic increase in chatter between iOS4 and ActiveSync.
 
Toxic,

I have the same set of problems, as do many who are using Exchange.

Apple has yet to call me back regarding mine but the guy I spoke with showed more and more issues popping up in the system regarding this and referred me to the forums.

I did the Reset Network Settings and it worked perfectly for about a day or so, today started having issues again.

BTW I just read your last post :)

Forget the kids on here and thanks for the help. I'll install this in a few minutes and see what happens.
 
ToxicMan said:
Asked and Answered.
There is no way to STOP users from upgrading to iOS4, as the option is presented to all users when they dock!
Click to expand...

A memo perhaps?

Could not predict the impact because we could not tell in advance that iOS4 handles activesync differently than prior versions which created rapid and significantly more traffic to ActiveSync services, which appeared as a result of the increased use of the new iOS.
Click to expand...

This is what all the developer builds are for. You could have tested it.
 
Not sure how much it helped....I just tried to send a 2.1 meg picture from my phone and it just sits there and spins...been 5 min so far.

Rebooted and it's still just hanging there....

Unfortunately I do not own my Exchange server, is there some more detail that you can pass on to why this happens so I can call my provider in the morning?

BTW, I posted about this about 5 minutes after upgrading to iOS4 on my 3gs, then again, twice, on the iPhone 4 only to get it pushed off the page.
 
beamer8912 said:
A memo perhaps?



This is what all the developer builds are for. You could have tested it.
Click to expand...

Testing iOS4 on one or even two units isn't going to create the same load as hundreds of devices across an enterprise will create on the ActiveSync infrastructure.

It's absolutely impossible to create the kind of load needed to replicate this situation in a test or development environment, and you're incredibly foolish if you think that testing this on one or two phones would have allowed the OP to identify the problem in test.
 
SHIFTLife said:
Testing iOS4 on one or even two units isn't going to create the same load as hundreds of devices across an enterprise will create on the ActiveSync infrastructure.

It's absolutely impossible to create the kind of load needed to replicate this situation in a test or development environment, and you're incredibly foolish if you think that testing this on one or two phones would have allowed the OP to identify the problem in test.
Click to expand...

I don't think its a load issue. FYI it took 12 minutes to send the 2.1mb file, then 5 seconds to receive it back. Tested my 3G speed and I'm way over 1mbps.

Back to the load issue. My Exchange problem started the minute I upgraded to iOS4. My provider is USA.NET and have a pretty stout infrastructure. I supposed I could check but I bet I was one of the first to upgrade. So the problems were there before the iPhone 4 was released.

My problems were immediate on the release date, if it slowly got worse then I could see that.

Through all of this Google says they have a fix in place since they use ActiveSync but have not bothered to share what they have fixed.
 
ToxicMan said:
UPDATE:
I was working with Apple over the weekend and have a patch for the iOS4 devices that extended the timeout settings on synchronization. We also added additional ActiveSync Servers which has solved the problem even for those that do not have the patch. It looks like the issue is still under investigation but from what we can tell so far, iOS4 made some changes to the way it interacts with Activesync and increased the traffic substantially. As a result, our infrastructure was overwhelmed by the additional traffic.

Those that posted on the Apple forums about reducing the size of the mailbox would have a similarly successful outcome because they're effectively reducting the load on the server. However, we opted to increase capacity by scalling out our activesync infrastructure.

We have also removed the patch from Apple on many of the affected devices and those devices continue to work well again without any additional issues.

Therefore, we have a temporary solution and will continue to work with applecare to determine the root cause in iOS4.
Click to expand...

My company is running a large iPhone pilot right now. I would say at this time at least 50 percent of the devices are running the 4.0 OS. We have made no changes and everything is working fine. We see no evidence of "increased traffic" or anything like that. It's just http/s traffic.
 
You rock SHIFTLife. Spot on true.

ERIC-Ptek. It's not an activesync problem in my professional opinion. There are changes in iOS4's implementation of activesync that are creating excessive chatter and increased traffic that are overwhelming the services and causing retransmits from the device which cause the sync to fail.

The patch at this point increases the timeout on the device from 30 seconds to four minutes. That is a temporary solution while we keep working the at the underlying problems. It allows the device to get as much data off the exchange hosts as it can before it time-out and fails causing it to retry again.

This will get solved, it's just a matter of time. We installed more activesync servers in the mean time to handle the additional load that iOS4 is causing. We used Vmware to spin up more serves and when we get it fixed with Apple, we will just de-provision the additional services.


You dont have control over the exchange service provider so you may just have to tough it out.
 
PhoneI said:
My company is running a large iPhone pilot right now. I would say at this time at least 50 percent of the devices are running the 4.0 OS. We have made no changes and everything is working fine. We see no evidence of "increased traffic" or anything like that. It's just http/s traffic.
Click to expand...

Super. Very glad it's not an issue for you. Sounds like you have enough headroom on your activesync servers. Or your not running exchange 2003.
 
ToxicMan said:
You rock SHIFTLife. Spot on true.

ERIC-Ptek. It's not an activesync problem in my professional opinion. There are changes in iOS4's implementation of activesync that are creating excessive chatter and increased traffic that are overwhelming the services and causing retransmits from the device which cause the sync to fail.

The patch at this point increases the timeout on the device from 30 seconds to four minutes. That is a temporary solution while we keep working the at the underlying problems. It allows the device to get as much data off the exchange hosts as it can before it time-out and fails causing it to retry again.

This will get solved, it's just a matter of time. We installed more activesync servers in the mean time to handle the additional load that iOS4 is causing. We used Vmware to spin up more serves and when we get it fixed with Apple, we will just de-provision the additional services.


You dont have control over the exchange service provider so you may just have to tough it out.
Click to expand...

Im a liittle confused about this. The only traffic between device and the active sync infrastructure should be http/s traffic and the heartbeat. Increasing the timeout interval would only change the amount of time between heartbeats I would think. This traffic is minimal. I'm not sure how the 4.0 os could increase activesync traffic
 
ToxicMan said:
Super. Very glad it's not an issue for you. Sounds like you have enough headroom on your activesync servers. Or your not running exchange 2003.
Click to expand...

The scary thing is our iPhone pilot has grown into a few thousand devices running off of an old ISA 2003 server. I'm to scared to touch it because everything seems to be working.
 
ToxicMan, just wanted to say I thought your story was awesome.

There are always things Apple (and anybody) can do to improve, and it was good to see they worked with you about it.

So thank you for not feeding the trolls and even showing the solution!
 
PhoneI said:
Im a liittle confused about this. The only traffic between device and the active sync infrastructure should be http/s traffic and the heartbeat. Increasing the timeout interval would only change the amount of time between heartbeats I would think. This traffic is minimal. I'm not sure how the 4.0 os could increase activesync traffic
Click to expand...

Not at all. So much more going on. Https is just a secure transport tunnel. The payload and server responses are what's of interest to us. I have a logging profile from apple that we are using to capture the communications between device and hosts. It's helping use look at the details within the packets and the responses coming back from the Microsoft hosts and how iOS4 is responding.

Isn't over, just a patch to get the affected enterprises through the night untill we get to the bottom of this.
 
You must log in or register to reply here.
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.
Back
Top