iOS4 and iPhone 4 Wreaking Havok in the corporation

Discussion in 'iPhone' started by ToxicMan, Jun 25, 2010.

  1. ToxicMan macrumors member

    Jun 22, 2010
    We have a very large mobile population in my company with thousands of apple iPhones and a growing population of iPads. We support a virtual desktop and mobile application suite.

    Starting Monday, our users began upgrading to iOS4 and on Wednesday and Thursday; many more started receiving iPhone 4.

    Beginning Tuesday, our helpdesk began taking increasing numbers of calls reporting intermittent loss of mail and calendar synchronization with our Microsoft Exchange services.

    By Friday, almost all activsync services are down affecting all our devices with iOS3 including the entire iPad population.

    There are some threads over on the apple forums suggesting a DNS change from "CNAME" to "A" type record for activesync (generically referred to in the threads as Exchange). However this has not turned out to be a solution to the issue.

    At this point, we have suspended adoption and are actively working to find a way back.

    Will keep this forum posted...
  2. beamer8912 macrumors 65816

    May 30, 2009
    This would do better in Tips, Help and Troubleshooting
  3. grantsdale macrumors 6502a

    May 11, 2008
    I suppose its possible. Tell them to fix their settings.
  4. klamse25 macrumors 6502a


    Oct 25, 2009
    It just so happens after every iPhone release many new people join our forums, (hence macrumors newbie title) and if a few people have a slight issue, they stretch it like crazy and troll the hell outta it.
  5. ToxicMan thread starter macrumors member

    Jun 22, 2010
    Thanks for the support, sort of. I typically don't post on fanboy sites, but because this is impacting so many of my users and nothing is showing up yet on general Google searches, I'm looking here and at apple forums.

    I suppose this is the wrong place given it's fanboy consumer user base and thus a "newbie" to the board is welcomed with open arms, NOT.

    It's this exact reason I tend to stay away from consumer forums and only contribute to professional association related user communities (like ASUG)

    But given the impact this is having within my shop, I'm looking everywhere for ideas (I have an open P1 case with Microsoft PS as well). Oh, and for those that don't know what Microsoft PS is, it's Microsoft Premier Support.

    Don't mean to be pedantic, but I'm in no mood right now for contributions that aren't helping.

    Please don't post if you have nothing worth contributing. I'm happy to chase down valid technical solution leads!!!
  6. xxSpudxx macrumors member

    Jun 17, 2009
    How much testing did you do?

    In an environment like that, you should have been testing all the way through gold brick release. If you did, why didnt this problem rear its ugly head then?

    Just curious.
  7. ToxicMan thread starter macrumors member

    Jun 22, 2010
    Couldn't agree more. Not happy with some folks who didn't get on the developers build and now I'm left to clean up.

    We just "reorganized" that group today... read between the lines.

  8. ToxicMan thread starter macrumors member

    Jun 22, 2010
    I was working with Apple over the weekend and have a patch for the iOS4 devices that extended the timeout settings on synchronization. We also added additional ActiveSync Servers which has solved the problem even for those that do not have the patch. It looks like the issue is still under investigation but from what we can tell so far, iOS4 made some changes to the way it interacts with Activesync and increased the traffic substantially. As a result, our infrastructure was overwhelmed by the additional traffic.

    Those that posted on the Apple forums about reducing the size of the mailbox would have a similarly successful outcome because they're effectively reducting the load on the server. However, we opted to increase capacity by scalling out our activesync infrastructure.

    We have also removed the patch from Apple on many of the affected devices and those devices continue to work well again without any additional issues.

    Therefore, we have a temporary solution and will continue to work with applecare to determine the root cause in iOS4.
  9. fuzion macrumors 6502

    Mar 15, 2010
    We have no issues that I am aware of w/ iPhone 4s and iOS 4 upgrades at my work.

    (I'm the IT guy so I would know about it)
  10. pugnut macrumors 6502a

    Jul 1, 2007
    Does your IT deparment not have an approved list of devices? Cleary they should have tested prior to allowing them. ISsues with active sync are not limited to the iPhone and are quite frequent with new devices that is why we test and approve.
  11. ToxicMan thread starter macrumors member

    Jun 22, 2010
    Asked and Answered.
    iPhone is approved device.
    There is no way to STOP users from upgrading to iOS4, as the option is presented to all users when they dock!

    Could not predict the impact because we could not tell in advance that iOS4 handles activesync differently than prior versions which created rapid and significantly more traffic to ActiveSync services, which appeared as a result of the increased use of the new iOS.

    We're up and working, and continue to work with Applecare and Engineering directly to see whats caused the dramatic increase in chatter between iOS4 and ActiveSync.
  12. Eric-PTEK macrumors 6502

    Mar 3, 2009

    I have the same set of problems, as do many who are using Exchange.

    Apple has yet to call me back regarding mine but the guy I spoke with showed more and more issues popping up in the system regarding this and referred me to the forums.

    I did the Reset Network Settings and it worked perfectly for about a day or so, today started having issues again.

    BTW I just read your last post :)

    Forget the kids on here and thanks for the help. I'll install this in a few minutes and see what happens.
  13. beamer8912 macrumors 65816

    May 30, 2009
    A memo perhaps?

    This is what all the developer builds are for. You could have tested it.
  14. Eric-PTEK macrumors 6502

    Mar 3, 2009
    Not sure how much it helped....I just tried to send a 2.1 meg picture from my phone and it just sits there and spins...been 5 min so far.

    Rebooted and it's still just hanging there....

    Unfortunately I do not own my Exchange server, is there some more detail that you can pass on to why this happens so I can call my provider in the morning?

    BTW, I posted about this about 5 minutes after upgrading to iOS4 on my 3gs, then again, twice, on the iPhone 4 only to get it pushed off the page.
  15. SHIFTLife macrumors 6502

    Jul 24, 2008
    Testing iOS4 on one or even two units isn't going to create the same load as hundreds of devices across an enterprise will create on the ActiveSync infrastructure.

    It's absolutely impossible to create the kind of load needed to replicate this situation in a test or development environment, and you're incredibly foolish if you think that testing this on one or two phones would have allowed the OP to identify the problem in test.
  16. Eric-PTEK macrumors 6502

    Mar 3, 2009
    I don't think its a load issue. FYI it took 12 minutes to send the 2.1mb file, then 5 seconds to receive it back. Tested my 3G speed and I'm way over 1mbps.

    Back to the load issue. My Exchange problem started the minute I upgraded to iOS4. My provider is USA.NET and have a pretty stout infrastructure. I supposed I could check but I bet I was one of the first to upgrade. So the problems were there before the iPhone 4 was released.

    My problems were immediate on the release date, if it slowly got worse then I could see that.

    Through all of this Google says they have a fix in place since they use ActiveSync but have not bothered to share what they have fixed.
  17. OttawaGuy macrumors 6502a


    Mar 28, 2006
  18. PhoneI macrumors 68000

    Mar 7, 2008
    My company is running a large iPhone pilot right now. I would say at this time at least 50 percent of the devices are running the 4.0 OS. We have made no changes and everything is working fine. We see no evidence of "increased traffic" or anything like that. It's just http/s traffic.
  19. ToxicMan thread starter macrumors member

    Jun 22, 2010
    You rock SHIFTLife. Spot on true.

    ERIC-Ptek. It's not an activesync problem in my professional opinion. There are changes in iOS4's implementation of activesync that are creating excessive chatter and increased traffic that are overwhelming the services and causing retransmits from the device which cause the sync to fail.

    The patch at this point increases the timeout on the device from 30 seconds to four minutes. That is a temporary solution while we keep working the at the underlying problems. It allows the device to get as much data off the exchange hosts as it can before it time-out and fails causing it to retry again.

    This will get solved, it's just a matter of time. We installed more activesync servers in the mean time to handle the additional load that iOS4 is causing. We used Vmware to spin up more serves and when we get it fixed with Apple, we will just de-provision the additional services.

    You dont have control over the exchange service provider so you may just have to tough it out.
  20. ToxicMan thread starter macrumors member

    Jun 22, 2010
    Super. Very glad it's not an issue for you. Sounds like you have enough headroom on your activesync servers. Or your not running exchange 2003.
  21. PhoneI macrumors 68000

    Mar 7, 2008
    Im a liittle confused about this. The only traffic between device and the active sync infrastructure should be http/s traffic and the heartbeat. Increasing the timeout interval would only change the amount of time between heartbeats I would think. This traffic is minimal. I'm not sure how the 4.0 os could increase activesync traffic
  22. PhoneI macrumors 68000

    Mar 7, 2008
    The scary thing is our iPhone pilot has grown into a few thousand devices running off of an old ISA 2003 server. I'm to scared to touch it because everything seems to be working.
  23. fizzwinkus macrumors 6502a

    Jan 27, 2008
    ToxicMan, just wanted to say I thought your story was awesome.

    There are always things Apple (and anybody) can do to improve, and it was good to see they worked with you about it.

    So thank you for not feeding the trolls and even showing the solution!
  24. ToxicMan thread starter macrumors member

    Jun 22, 2010
    Not at all. So much more going on. Https is just a secure transport tunnel. The payload and server responses are what's of interest to us. I have a logging profile from apple that we are using to capture the communications between device and hosts. It's helping use look at the details within the packets and the responses coming back from the Microsoft hosts and how iOS4 is responding.

    Isn't over, just a patch to get the affected enterprises through the night untill we get to the bottom of this.
  25. sdsvtdriver macrumors 65816

    Jul 31, 2008
    Southern California

Share This Page