Extended MobileMe Email Outage Due To Server Issue

[TPALTony]

Seconded

Your figure of 0.0001% means uptime of 99.9999% - no one achieves that.

Having RAID and a decent backup system is only a small part of what you need to achieve high availability - non interruptible power supply, full hardware and network redundancy, fallback sites and so on.

Again, expecting or trying to achieve a high level of availability for what is a budget web hosting service is just plain stupid.

I completely and utterly agree with you. People think that uber uptime is trivial, and it simply is not. "5 nines" is the pinnacle, which is 99.999% uptime. If you do the maths here, you get 53 MINUTES of downtime. 5 nines is always quoted against the full year, not "unscheduled downtime." That would mean that all upgrades you ever wanted to do, plus an allowance for some unexpected downtime had to fit into 53 minutes a year. I design and build trading systems and I know from first hand experience that each nine is exponentially harder than the last to achieve. The fifth nine is _incredibly_ hard, and _incredibly expensive._ For a trading system, we do it. For a $99 a year consumer service, no way. You wouldn't make money given the costs. When you shoot for 99%, you code for reliability. When you shoot for 99.999%, you are incredibly hand tied in terms of architecture. You have to be able to bounce one node of a cluster without bringing them all down, and handle the associated issues with the fact that the code bases will be different on different nodes for a short time. That's not easy to handle and you end up putting a TON of time into handling a time window that if you could only take all the servers down for an hour, you wouldn't have to cope with. The difference between 99% and 99.999% is night and day. And 99.9999%. That doesn't exist. If it does, it's not economically viable.

99% uptime, on the other hand means you can have 3.65 DAYS of downtime, which is probably OK, given maintenance, and given that "down" usually means totally, whereas mobileme, like many things, is a series of services that don't all have to come down at the same time...

For a consumer service, 99% seems about right. Which of course, does not excuse the fact, that right now they have blown that completely too, at least for email. They can fudge the statistics though. Mail is down, but everything else is working, so is it 5 days of downtime for mobileme, or 5 days adjusted for it just being mail...

be well

t

 

[BongoBanger]

MobileMe = Vista

Yup, initial difficulties and 99% of people who use it are satisfied.

Seriously though, this is a banana skin for Apple. Although 1% doesn't sound much it means that 1% of people are getting no service at all. This isn't the same as, say, a temporary outage that effects a lot of users for a short period of time.

The problem is that Apple market themselves as a high quality provider so when mistakes like this are made the effect is somewhat amplified.

 

[javierbds]

"Web 2.0" woes

you guys think this hurts apple's image?

I guess that's a rethoric question ... Indeed, but not too much, Apple makes this kind of mistakes with version 1 products (most companies do). Apple gets them right after some time (others don't).

What it hurts (I think) is the idea of reliable, secure, always-on, featureful on-line sw: That all that ajax-js, web services, browser architectures can really substitute all desktop and server archs ... SaaS is not that easy to get right (maybe a Web 2.0 bubble is on the works) With the excuse that the client does not have to install sw manually, you patch like hell ... And then you can blame the client internet provider
Traditional C/S architectures CAN and DO achieve all those 9s in availability and with a high QoS (with redundancy, multiple sites, rerouting, queues, planned staged upgrades ...). I'm not so sure you can achieve that riding the web and for a consumer service ...

Also, maybe Apple is sending the message some of its latest sw is not up to par ... (released too fast, too many platforms (Desktop, Mobile, Server, Web ...), too little testing). IMHO it is not worth it to be early if you have to repeatedly patch, but somehow consumers cope with it. I rather prefer a less featureful but working sw piece, to be upgraded later, than one where most functionality is already there but it sometimes fail ...

 

[8CoreWhore]

The host: http://en.wikipedia.org/wiki/Akamai_Technologies

 

[diamond.g]

To achieve this reliability figure all you need is reliable RAID, daily backup of all data, and offsite backup on a near daily basis.

This is established technology and procedure.

If Apple is not doing that, they are overextended. One wonders if this could be solved by having a co-location agreement with Google, who has similar problems, if on a larger scale.

One also wonders if the adoption rate is actually taxing the bandwidth capacity of their new (former Worldcom) Telco site and their regional provider switch(es).

Rocketman

Apple should be using a provider that has at least an OC48 level of connectivity. Shoot Apple is large enough to warrant multiple OC96/192 lines.

MobileMe is really Exchange for the rest of us. Apple's new MobileMe slogan should be, "The reliability of exchange, now in Apple goodness."

I am not sure if you are claiming exchange isn't reliable. If so that is a pretty inaccurate statement. You would find that if exchange were not reliable big business wouldn't use it and Apple wouldn't not have included compatibility with it in the iPhone nor Snow Leopard.

 

[xnetco]

The host: http://en.wikipedia.org/wiki/Akamai_Technologies

These aren't the hosts. They just deliver some of the content.

 

[avn]

Make the service free, and call it beta

Alright, here is what I think:

It probably wouldn't be a bad idea for to make the service free, call it a beta, and spend some more money/time in developing it. This clearly hasn't been done yet.

If the product is free and classified as beta:
1) smart people should not be duped into using the application for mission-critical systems.
2) there would be more consumer support for its development (hey apple is working on this great web-based contact/file server!).
3) it would be the right thing to do. I say this because I just suffered a huge feature loss when I migrated to the iPhone. Currently MobileMe is the only way to wirelessly sync my calendar with my iphone (I cannot believe I lost this feature when I transitioned from my Nokia!!!).


This has the potential of becoming one of the killer-apps, that would increase the visibility of mac as a leader in tech innovation, but only if it works.

 

[Shasterball]

I don't know enough about this so I am confused as to how a server can be down for so long? I'm guessing it's not an equipment problem because that can be swapped out. Is it software?

 

[idea_hamster]

More wide-spread than acknowledged

I believe this e-mail outage is more serious and affecting more people than Apple is acknowledging. Here's a transcript (edited for relevance; emphases added) of a chat with MobileMe support:
**********
[idea_hamster]: My e-mail only add-on account has been off-line since Friday. What's up with that?

: Yes we are currently experiencing a outage affecting about 10% of our email addresses. At the time we do not have a exact time when it'll be resolved but should be somewhere around 6 a.m. tomorrow

[idea_hamster]: Ten percent? The help page says 1%. Typo? Or is the problem more wide-spread?

: Yes it started out at 1% but we found out a whole mail store was bugged. There is something causing the mail to not be delivered and bounced back

[idea_hamster]: Well, I guess I can just wait and hope that it's fixed by tomorrow....

: Yes I 've been looking for any kind of work around because unfortunantely my mail was affected also but I have been unable to find anything that helps

[idea_hamster]: Are we going to get a pro-rated refund on our mobileme.com membership charge? Or a free song from iTunes? Or a handful of mud in the face and a kick in the head? ;-)

: A pro-rated refund is unlikely unless the outage continues for like another week, and we can't give iTunes products from mobileme so we're probably out of luck

[idea_hamster]: No problem -- I understand. Thanks, though. So it's the mud and the kick then!

[idea_hamster]: Thanks for your help!

: Thank you for being so understanding David!
**********

Apple has consistently referred to 1% of e-mail addresses being affected, but clearly some people at Apple consider this to be an issue for up to 10% of addresses.

I think that Apple realizes that a 1% outage sounds like "growing pains" but a 10% outage sounds like Vista.

BTW, as of 6:00 AM today, my mac.com address has been "affected" for five days. Still not fixed.

 

[puckhead193]

Wirelessly posted (my blackberry pearl: BlackBerry8130/4.3.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/105)

At least they are sending out emails most companys won't even do that.

 

[paulg]

My account is once again working fine but has been out for about 3 days.

 

[Laglorden]

MobileMe = Vista

Nah, got to be MobileMe = Windows ME

Then MobileXP = Windows XP before MobileVista = Windows Vista.

So, they're still a couple of failures behind.

 

[Bubba Satori]

you guys think this hurts apple's image?

If they fix it in the next couple of days, no.

 

[swingerofbirch]

I wonder if Apple knows about this company that sells servers:

http://www.apple.com/xserve

 

[brn2ski00]

I had issues with my MobileMe Mail all day yesterday.... Sent mails weren't being sent, I wasn't receiving mails on time, it stinks when you rely on it for business.

 

[LarryB08]

Nah, got to be MobileMe = Windows ME

Then MobileXP = Windows XP before MobileVista = Windows Vista.

So, they're still a couple of failures behind.

I am in both "camps" as, for me, there is a lot to love about both Macs and PCs. But I just had to say that anyone who thinks that Windows XP is/was a failure is seriously deluded by all that "Apple goodness."

 

[fastbite]

My partner has been 7 days with out Mail. And yes, she's set up Googlemail now but still, what a joke.

 

[diamond.g]

I wonder if Apple knows about this company that sells servers:

http://www.apple.com/xserve

Wouldn't Apple have to run MobileMe on much beefier boxes than some 1U 8 core rigs? I would think they would need tons of 16 core boxes...

 

[mspman]

I'm one of the illustrious 1% (or 10%.. who knows). But my problem is limited to sending email.. I can receive email.. but anything I try to send goes into la la land.

Personally, I think someone at Apple horked up something. There was a backup missing, or a disk was dropped that didn't have a current backup.. something like that. I work in IT, as I'm sure a bunch of us do too, and if I had an outage like this it would be my fault and I would be fired. This is 2008 - I'm sure that if it's a bad server that it can be replaced or repaired.. seen it a million times. There's a human element to this that we'll never know about..

 

[JC4]

I'm done

My account expires in August, and I just turned off auto-renewal. I've only hung on this long because of promised improvements and new features. I just don't see anything worth paying for. The only thing I really use it for is syncing my macs, and that's been less than reliable, especially since MM. On top of that, performance still sucks(its slow).

Bye .mac/mm.

 

[zelet]

I'm waiting for Google to release Picasa for Mac (photo sharing) and Snow Leopard then I'm done with MobileMe. I've had my fill (dotMac has sucked for a long time too).

EDIT: I found out Google has a Picasa Web Upload plugin for iPhoto. I'm that much closer to abandoning dotMac! Just need a way to sync Calendars.

 

[Mindflux]

Just need a way to sync Calendars.

SpanningSync or BusyMac.

Now you just need something that'll push those changes real time to your iPhone

 

[msw123307]

i haven't really had problems with email. but since last night i keep getting this message saying "server connection interrupted" showing my .mac name. what is this? i have no idea.

I was originally just closing the dialog box it came in, but now i hit disconnect. i just have no idea what i've disconnected.

 

[edwardlanti]

I haven't been able to send or receive email since I made the account 6 days ago

 

[coleridge78]

I'm waiting for Google to release Picasa for Mac (photo sharing) and Snow Leopard then I'm done with MobileMe. I've had my fill (dotMac has sucked for a long time too).

EDIT: I found out Google has a Picasa Web Upload plugin for iPhoto. I'm that much closer to abandoning dotMac! Just need a way to sync Calendars.

Eh, good luck. GMail has more downtime, data loss, and data exposure flaws than any other "big name" mail service. And they give away your info to anyone who'll give them money, to boot. I just don't get the infatuation... their search is great. They're a bunch of math nerds. Everything else they do is buggy, insecure, and nearly unusable unless you have very low standards.