Fastly is bleeding millions of dollars a second for this. Would not want to be any engineer even remotely responsible for their services. Wow.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Fastly CDN Server Issue Brings Down Swathes of Websites [Update: Solved]
- Thread starter MacRumors
- Start date
- Sort by reaction score
Seems like again BBC News all fine in the UK. Think it's anything on the bbc.co.uk URL which is fine, versus the commercial/worldwide bbc.com which is down.
Yea can imagine all the financial penalties built into the SLA/contracts. Ouch.
Because what passes for 'good architecture' in modern ops/infra practices tends to be a list of tickboxes with "are we using an external service provider and thus don't need to hire anyone experienced in this technology?" next to them all.
This Twitter thread from the Guardian's technology reporter is pretty useful in understanding what's going on:
Sadly humans being human, it’s exactly this sort of thing that people spread with: “MacRumors said it might be a Cyber attack”.
Let's do an experiment:
You hire "experts" to keep your services up worldwide across multiple regions, with sub-second performance, and build tools for your software developers to easily build applications on top of. They would have to set up the services in tens, if not hundreds of physical server locations across the world as well.
I use Fastly.
Let's see how often your "experts" f up in house vs Fastly and how much more money your "experts" cost and how much your software developers hate the in-house system.
I love how every time this happens, which is rare, someone with zero knowledge of scaling at this size comes in and writes how companies are lazy or cheap or both.
I love how every time I point out that relying on single external vendors for something is a problem, someone with zero ****ing clue about the topic - in this case infrastructure in general, or load balancing and failover in particular - gets an internet boner about telling me I'm wrong.
They are stating their belief which I agree with. There have been an alarming number of hacks across the last few years that were originally reported as something else to mitigate damage to the stock value likely.
I recent weeks it’s become clear that companies have been getting highjacked for sometime and paying ransoms without public disclosure of the attacks.
Let's be real, if you were actually good at those things, you wouldn't be living in Thailand and every other posts you make here is cussing people out. These companies including Paypal, Spotify, Twitch, etc. would be clamoring to hire you. They're not.
PS. I manage services for a popular product that receives ~2-3 billion visits/year. I think I know a thing or two about CDNs and fallbacks.
All it takes is for it to happen once and then it can happen again and again. What this has shown is just how much of a cavalier attitude website developers have when designing a site. The core code of the site is on a hosting sites main server but everything else, images, videos, scripts, java, cookies, other web links, all come from other servers dotted around the world using the cloud and distributed by a cloud based CDN.
When you open a website, take a look in the bottom left hand corner of the browser and notice just how many different locations the website connects to, in some cases it can be as high as 30 before the website displays as it should do.
When you open a website, take a look in the bottom left hand corner of the browser and notice just how many different locations the website connects to, in some cases it can be as high as 30 before the website displays as it should do.
Why? Because managing infrastructure and creating dev tooling requires me to sit at a desk in only one specific country?
I've been working on infrastructure projects since we had remote student sites connecting over 64K ISDN lines mate, and "well **** what happens if that single X goes down" was a lesson learned well before I was in a role that meant making final decisions about things.
Good, im glad your business is doing well.
Lots of people think they know lots of things. Whether they do or not is sometimes a different story.
Which amazon site was affected? (Not being facetious I don't really use most of the sites listed in the original post so I don't know which of them may be Amazon owned)
Pretty sure this outage violates any “5 nines” agreement as the outage was longer than 6 minutes or whatever.
We don't use Fastly so I don't know, but from a quick look at their website the only mention of uptime guarantees is on the "gold" support plan (which is the first level above 'standard', but below 'enterprise') and it indicates a 100% uptime guarantee, but doesn't talk about outage credits or reimbursals etc.
We need web3, we need IPFS, we need decentralization. Web2 has been co-opted by surveillance. Apple are doing a sterling job on privacy, but that's one company trying to protect their customer from the ills of the centralized internet. It needs to be baked in.