Duke Network Problems from Cisco, Not iPhone

[MacRumors]

Earlier this week, a story came out that Duke university was having major wireless network issues and blamed the Apple iPhone. Several parts of the story didn't quite add up, and as it turns out in a statement today, that this issue was in fact a Cisco issue:

Cisco worked closely with Duke and Apple to identify the source of this problem, which was caused by a Cisco-based network issue. Cisco has provided a fix that has been applied to Duke's network and there have been no recurrences of the problem since. We are working diligently to fully characterize the issue and will have additional information as soon as possible. Earlier reports that this was a problem with the iPhone in particular have proved to be inaccurate.

Article Link

 

[KindredMAC]

Now if only the major news outlets would run this story as much as they ran the negative iPhone spin of it.

My friend told me today that his wife has "heard all this bad news stuff" about the iPhone. When she was asked what it was that she heard, she couldn't remember the details.... Sounds like the good old "MACS ARE BAD!" crap from the 90's is coming back and has attached itself to the iPhone.

 

[Peace]

Now if only the major news outlets would run this story as much as they ran the negative iPhone spin of it.

My friend told me today that his wife has "heard all this bad news stuff" about the iPhone. When she was asked what it was that she heard, she couldn't remember the details.... Sounds like the good old "MACS ARE BAD!" crap from the 90's is coming back and has attached itself to the iPhone.

This will never happen because I personally believe the news media/Rupert/Microsoft conglomerates are out to get Apple.

Expect more dis-information over the next 2 years as Apple slowly takes over.

P.S. this article should be dugg

 

[massiv]

Right on Peace

This will never happen because I personally believe the news media/Rupert/Microsoft conglomerates are out to get Apple.

Expect more dis-information over the next 2 years as Apple slowly takes over.

P.S. this article should be dugg

I think you hit the nail square on the head. I actually just wrote eDuke's editor voicing my frustrations (tactfully). The dis-information is going to keep piling on.

 

[Lancetx]

More to the point, Kevin Miller should be the one called out here more than anybody.

Kevin Miller, assistant director, communications infrastructure, with Duke’s Office of Information Technology, laid the blame for Duke’s networking problems squarely on the iPhone.

The network team began capturing wireless traffic for analysis and that’s when they discovered that the offending devices were iPhones, Miller said earlier this week.

“I don’t believe it’s a Cisco problem in any way, shape or form,” Miller said firmly.

It would appear at this point that Miller made his statements without a full report or any basis to blame the iPhone or Apple for the problems.

 

[Camui]

Lancetx~~ I completely agree with you.
That was completely unprofessional of him to have stated that he believed the iPhone was the problem. From what I understand, he did that before they'd actually tried finding out what was causing it. That is just stupid. You never go on record with your personal opinion, or even just on record, before having more information. Not to mention representing your business like that, unless you state that whatever you are saying isn't the belief or opinion of your business, is also complete idiocy. Not only does it make you look bad but it makes whoever you work for look REALLY REALLY BAD.

I wonder if this gentlemen is Anti-Apple....or just thinks he's A LOT smarter than he really is.....

 

[Fuchal]

Lancetx~~ I completely agree with you.
That was completely unprofessional of him to have stated that he believed the iPhone was the problem. From what I understand, he did that before they'd actually tried finding out what was causing it. That is just stupid. You never go on record with your personal opinion, or even just on record, before having more information. Not to mention representing your business like that, unless you state that whatever you are saying isn't the belief or opinion of your business, is also complete idiocy. Not only does it make you look bad but it makes whoever you work for look REALLY REALLY BAD.

I wonder if this gentlemen is Anti-Apple....or just thinks he's A LOT smarter than he really is.....

Usually when someone thinks that they are a lot smarter than they really are, they are anti-apple as well.

 

[ctakim]

This speaks volumes about the quality of the Duke IT system more than anything to do with Apple.

 

[mustang_dvs]

This speaks volumes about the quality of the Duke IT system more than anything to do with Apple.

A fair amount of the IT staff at Duke are, in fact, Apple users.

 

[peterjhill]

More to the point, Kevin Miller should be the one called out here more than anybody.

The one that should be called out are all the people who tried to capitalize on the problem to write a negative iphone story to get hits on their website.. I blame Network World.

Here is Kevin's original post that started this whole thing

http://listserv.educause.edu/cgi-bin/wa.exe?A2=ind0707&L=wireless-lan&P=2182

Read it before you blame him. He does not say that it is a problem that the iPhone needs to fix. He says that the problem seems to be related to the iPhone (it was a Cisco bug triggered by the iPhone, but something that Cisco will need to fix).

Network world is the one that took Kevin's post on Educause, a perfectly legitimate place to post such a message, and then it got dugg, then the story went viral. Kevin was not trying to become famous, he was trying to fix a problem.

If Kevin could have done anything different, I am sure that he would have ignored Network Worlds request for information and just kept on working on finding the root cause.

Kevin is a brilliant software developer who knows more about TCP/IP than most Network Engineers. He owns an iMac along with a number of other Linux and Windows machines. I know Kevin personally and wish that people who do not know the entire context of this story would stop blaming him personally. Please cut him some slack. Do a google search for "Kevin Miller -iphone (to get rid of any links with iphone) and you can see some great examples of his work.

 

[strangelogic]

Please cut him some slack. Do a google search for "Kevin Miller -iphone (to get rid of any links with iphone) and you can see some great examples of his work.

It isn't like he made up some story about WMD's and started a war that continues to de-stabilize the middle-east and thus the world. However, he provided quotes that pretty much stated unequivocally that as such an 'expert' he couldn't see this being a Cisco bug...

Brilliant technical people often fall into this trap when pressured to resolve/debug a problem, and when their gut-instinct (the new device in this picture is the iPhone thus it must be the 'problem') is wrong there is a price to pay. A certain degree of infamy is a healthy lesson - a growth experience.

He will be fine, and hopefully there will be some 'responsible' press - at least the financial reporters that clear up this story to a larger audience...

 

[peterjhill]

Strangelogic, that was a very reasonable post, and I appreciate it. I am not sure that any Cisco is better than Apple thing played any part here. The problem with the Lightweight APs is that they are creating tunnels between the controllers and the APs and it makes it harder to sniff traffic to figure out what is going on.. If you have an AP that that replicating the ARP requests due to some bug, it will be hard to identify if the ARPs are coming from the client or the AP. You can't just sniff the wired uplink to the AP.

I shared an office with Kevin a few years ago and he is an amazing developer. While he was still a student at CMU, he wrote the best DHCP/DNS network registration/management tool (www.net.cmu.edu/netreg). As a staff member he wrote CMUs automated IDS/abuse tool (epidemic). His software did what it takes literally 15 people in my last two jobs to do (amazon.com and University of Washington).

I hope it does blow over soon. "nothing to see here" ;-)

 

[ukneeq]

The one that should be called out are all the people who tried to capitalize on the problem to write a negative iphone story to get hits on their website.. I blame Network World.

Here is Kevin's original post that started this whole thing

http://listserv.educause.edu/cgi-bin/wa.exe?A2=ind0707&L=wireless-lan&P=2182

Read it before you blame him. He does not say that it is a problem that the iPhone needs to fix. He says that the problem seems to be related to the iPhone (it was a Cisco bug triggered by the iPhone, but something that Cisco will need to fix).

Network world is the one that took Kevin's post on Educause, a perfectly legitimate place to post such a message, and then it got dugg, then the story went viral. Kevin was not trying to become famous, he was trying to fix a problem.

If Kevin could have done anything different, I am sure that he would have ignored Network Worlds request for information and just kept on working on finding the root cause.

Kevin is a brilliant software developer who knows more about TCP/IP than most Network Engineers. He owns an iMac along with a number of other Linux and Windows machines. I know Kevin personally and wish that people who do not know the entire context of this story would stop blaming him personally. Please cut him some slack. Do a google search for "Kevin Miller -iphone (to get rid of any links with iphone) and you can see some great examples of his work.

I definetly blame him. He said he didn't think it was a cisco problem in any shape or form. He should be held accountable for his words!

 

[X86BSD]

Idiots in charge, what a shocker.

And this is why I left IT. 95% of IT workers couldn't tell an ARP from a Carp.
Too many asshats in charge who don't know clue one about anything. It's the 5% with a clue that have to keep running around fixing the poor decisions of clueless management and bumbling oaf co-workers. I actually worked with a CCSP guy that completely blocked ICMP on his precious little Sun firewall server for the corporate network. And when I approached him about it to try and explain to him why you cannot just slap an embargo on the entire ICMP protocol because you break things like P-MTUD, at first he didn't know why he should undo his icmp filter, then when I explained things to him he just said and I quote "So what? No one uses it anyway!" He was Cisco certified for gods sakes. And after 15 years of hearing idiots all around me say stupid things I thought I heard it all. Wrong. It's a shame really, to see that the state of IT hasn't changed.

X

 

[ctango]

One should never make a judgement unless the facts have been clearly laid out. Kevin saw an iPhone and considered it the problem. He did not research, nor did he assign someone enough time to research the problem thoroughly. Had he waited another day or two for more detailed information on the problem, the players in the problem, and the possible causes, none of this would have made any media.

 

[peterjhill]

One should never make a judgement unless the facts have been clearly laid out. Kevin saw an iPhone and considered it the problem. He did not research, nor did he assign someone enough time to research the problem thoroughly. Had he waited another day or two for more detailed information on the problem, the players in the problem, and the possible causes, none of this would have made any media.

I would turn your statement inside out.. the people that are personally blaming him should not make any judgments until they understand the facts.

He was in the middle of researching the problem. His name got tied to this because he was researching the problem. Unless you can look at his initial educause post and understand it, then it is hard for me to imagine how one can judge him.

He was not the one that contacted the media about the story. They contacted him.

Anyone who understands the complexities of troubleshooting lightweight access points would see the challenge. These APs make a GRE tunnel back to a central controller. One cannot just sniff the iphone and check the rate it is sending arps, because the thing you are using to sniff is a potential cause, and in fact was part of the problem.

Kevin is certainly not a clueless leader, I know him personally. He graduated with honors with a CS degree from Carnegie Mellon... one of the top two schools in the field. I am sure that when the wireless network started having such serious problems, there was a huge effort to find out what was breaking.. Kevin did not time the questions he received from Network World. He is not the one that blew this out of proportion.

Anyone who actually has ever had to solve problems like this will know it is a complicated process and that interrupting engineers for interviews is not going to make the problem get solved any faster.

 

[massiv]

I would turn your statement inside out.. the people that are personally blaming him should not make any judgments until they understand the facts.

He was in the middle of researching the problem.He is not the one that blew this out of proportion.

Anyone who actually has ever had to solve problems like this will know it is a complicated process and that interrupting engineers for interviews is not going to make the problem get solved any faster.

"My concern is how many students will be coming back in August with iPhones? It’s a pretty big annoyance, right now, with 20-30 access points signaling they’re down, and then coming back up a few minutes later. But in late August, this would be devastating... It may have something to do with the iPhone losing connectivity and then trying to reconnect in a new location... I don’t believe it’s a Cisco problem in any way, shape, or form." -

“I’m not exactly sure where the ‘bad’ router address is coming from,” Miller says. One possibility: each offending iPhone may have been first connected to a home wireless router or gateway, and it may automatically and repeatedly be trying to reconnect to it again when something happens to the iPhone’s initial connection on the Duke WLAN."

“I don’t believe it’s a Cisco problem in any way, shape, or form,” he says firmly.

Is that not his statement?

Nice approach.

He said what he said. i can appreciate your stance but it is what it is. You cannot retract published statements anymore, even in context =)

 

[ctakim]

yes, I am sure that Kevin is a nice guy and loves his dog and his Macs. But it does not change the fact that he came off as a total and complete incompetent moron. Is that fair, probably not, but he should choose his words more carefully and should know the risks when making statements to the media. Are you saying this his statement exonerating Cisco was never made?

A fair amount of the IT staff at Duke are, in fact, Apple users.

And your point? Whether a Mac or a PC sits on your desk has no significant bearing on how competent you are at your job.

 

[mrowl]

I would turn your statement inside out.. the people that are personally blaming him should not make any judgments until they understand the facts.

He was in the middle of researching the problem. His name got tied to this because he was researching the problem. Unless you can look at his initial educause post and understand it, then it is hard for me to imagine how one can judge him.

He was not the one that contacted the media about the story. They contacted him.

Anyone who understands the complexities of troubleshooting lightweight access points would see the challenge. These APs make a GRE tunnel back to a central controller. One cannot just sniff the iphone and check the rate it is sending arps, because the thing you are using to sniff is a potential cause, and in fact was part of the problem.

Kevin is certainly not a clueless leader, I know him personally. He graduated with honors with a CS degree from Carnegie Mellon... one of the top two schools in the field. I am sure that when the wireless network started having such serious problems, there was a huge effort to find out what was breaking.. Kevin did not time the questions he received from Network World. He is not the one that blew this out of proportion.

Anyone who actually has ever had to solve problems like this will know it is a complicated process and that interrupting engineers for interviews is not going to make the problem get solved any faster.

So Kevin didn't say this?

“I don’t believe it’s a Cisco problem in any way, shape or form,” Miller said firmly.

and FYI, just because you graduate with "honors", doesn't make you a good leader, book smart, yes, but leadership, no.

 

[york2600]

Where's the fix?

Does anyone have a link to the fix? We just tested out an iPhone the other day on our network here at Humboldt State University and the APs dumped on us after the ARPs. We'd love to get this fixed before students show up in a few weeks.

 

[Peace]

Does anyone have a link to the fix? We just tested out an iPhone the other day on our network here at Humboldt State University and the APs dumped on us after the ARPs. We'd love to get this fixed before students show up in a few weeks.

I'm guessing you work in the IT Dept. ?

If so I'd say talk to whoever you deal with rather than ask on a message board.

No offense intended.

 

[MacAerfen]

The problem I see here is comments like "The media came to him" as an excuse. He did start out by trying to solve the issue. His post was not the problem, and people are not frustrated at him because of his post, which was a reasonable post aimed at trying to resolve an issue. The crux of the issue is that when contacted by media instead of being honest he allowed ego to dictate his response. The fact was that he had not yet identified the actual cause, and in fact allowed prejudice to blind him to the truth. He ruled something out arbitrarily. Regardless of how brilliant he may be, he failed the most basic troubleshooting policy: Until its been checked, don't rule it out. Your gut instinct might be that theres nothing wrong with the router. That does not rule it out and it leads you to make mistakes. From a troubleshooting perspective the first thing checked should always be the one part you are 99% sure is not the cause. That way you have a solid base from which to troubleshoot, and it does not come back to bite you in the ass.

What he did was very irresponsible when contacted by the media, and he needs to be out there trying to get published on wherever he can to appologize for it. He needs to be generating as much press as possible telling people the truth to compensate for his error.

When contacted by the media he should have said "At this point it we do not have an exact cause, we are working hard to identify the issue and resolve it. Right now we are seeing information coming from the iPhones and are trying to gather more information to determine if they are responsible, but are looking into all aspects of it." That provides an accurate statement of what is going on. It also allows him to alert others similarly researching the issue on something to look at. There is no harm in suggesting other network officials check into the iPhone, as long as you are clear that you have not isolated it to that device. The idea is to generate awareness so more people are looking into finding the answer. The harm comes when you make generalizations and plant blame on something without proof so you sound intelligent.

 

[Rodimus Prime]

wow talk about some fan boys.

Lets look at some simple facts here on why the conclusion that it was apple fault.

First the iPhone was the devices struggling with their wireless network. Other devices where working just fine. That be laptops or window model devices. Palm I would suspect would not work on their network due to limitation in palm wireless software (palm fault)

Yet people here can not understand that fact. It a fact that the bug was caused by the iPhone not some other devices but the iPhone. When it ONLY the iPhone not working with the network then you are right the iPhone is going to eat the blame and any intelligent person should understand that fact well unless you are a fan boy or apple then well it still a true statement because intelligents goes out the window and you are no longer an intelligent person.

 

[Peace]

wow talk about some fan boys.

Lets look at some simple facts here on why the conclusion that it was apple fault.

First the iPhone was the devices struggling with their wireless network. Other devices where working just fine. That be laptops or window model devices. Palm I would suspect would not work on their network due to limitation in palm wireless software (palm fault)

Yet people here can not understand that fact. It a fact that the bug was caused by the iPhone not some other devices but the iPhone. When it ONLY the iPhone not working with the network then you are right the iPhone is going to eat the blame and any intelligent person should understand that fact well unless you are a fan boy or apple then well it still a true statement because intelligents goes out the window and you are no longer an intelligent person.

Did you even read the news?

and it's intelligence.

 

[peterjhill]

Did you even read the news?

and it's intelligence.

We all appreciate your spell and grammar checking. How much do you charge?

As for the problem being triggered by the iPhone, that is true. The iPhone may not have been doing anything out of spec for the RFC, but it was doing something different than other hosts on the wireless network.

Most hosts flush their ARP cache whenever their network connection drops. The iPhone is seems to use previous ARP information to figure out if an SSID matches a previously connected wireless network. This is different than how most hosts work. Most hosts will associate with a wireless network, DHCP, then ARP for the default gateway. The iPhone is ARPing for a gateway that is not on the network and not being handed out by DHCP. That is different behavior.

These LWAP wireless networks are complicated. The configuration of them is much more complex than your home AP. We are talking about multiple SSIDs each with their own security policies. 802.1q trunked uplinks or GRE tunnels back to central controllers. A problem like this take quite a bit of effort to troubleshoot. Kevin was stuck in a bad position as being a person who legitimately could talk to the press, as an Associate Director, as well as probably the smartest Network Engineer there, even though that is not his title.

He, also, did not use the word "firmly" that Network World put into the article. That was something the writer put in as an interpretation of whatever interview they did. We also don't have the entire interview, do we. We do not know what lines the press cherry picked from the interview do we? No we don't. Do we know if there was a, "from what we have check so far" before the "It does not seem to be a Cisco problem" or whatever the quote was.

I also get peeved at the people who are trying to tie this to the Duke rape case. They are completely unrelated. Anyone who thinks or suggests they are is daft.