Cloudflare Bug That Leaked Sensitive User Data From Various Websites and Apps Now Fixed

[MacRumors]

Content delivery network Cloudflare has confirmed the existence of a bug that caused search engines to cache sensitive user data from a variety of well-known apps and websites. Google researcher Tavis Ormandy discovered and reported the bug to Cloudflare, and the company has since fixed the bug and published a detailed blog post about exactly what happened.

According to Cloudflare, the period of greatest impact for the "parser bug" ran from February 13 to February 18, although the extent of the leak stretches back months. The heart of the issue was a security problem with Cloudflare edge servers, which were returning corrupted web pages by some HTTP requests running on Cloudflare's large network.

In what the company referred to as "some unusual circumstances," occasionally private information was returned as well, including "HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data."

It turned out that in some unusual circumstances, which I'll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.

The bug was serious because the leaked memory could contain private information and because it had been cached by search engines. We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence.

As shared in a tweet by Ormandy this week, that data also included private dating site messages from OKCupid, full messages from a "well-known chat service," passwords from password managing apps like 1Password, and more (via Fortune). In response, some companies -- like 1Password -- have published blog posts confirming that "no 1Password data is put at any risk through the bug reported about CloudFlare."

To expedite a solution, Cloudflare responded to Ormandy's discovery and turned off three minor features of the network -- email obfuscation, Server-side Excludes, and Automatic HTTPS Rewrites -- discovered to be using the same HTML parser chain "that was causing the leakage."

In its blog post, the company said that it has "not discovered any evidence of malicious exploits" in relation to the time that the parser bug was active. It also noted that, while serious, the scale of the bug was still relatively low: around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulted in memory leakage. "That's about 0.00003% of requests," the company noted.

Cloudflare worked with the affected search engines, including Google, Yahoo, and Bing, to erase any remnants of the sensitive data from their caches. The company's chief technology officer, John Graham-Cumming, concluded the blog saying, "We are very grateful to our colleagues at Google for contacting us about the problem and working closely with us through its resolution. All of which occurred without any reports that outside parties had identified the issue or exploited it."

Earlier this week, it was reported that Apple cut ties with server supplier Super Micro Computer in order to avoid a potential future scenario where user data might be put at risk, similar to Cloudflare's leak. Early in 2016, Apple was said to have discovered a potential security vulnerability in one of Super Micro Computer's data center servers and effectively ended its business relationship with the network company shortly thereafter.

For a technical dive into Cloudflare's parser bug and its origins, check out the company's blog post.

Article Link: Cloudflare Bug That Leaked Sensitive User Data From Various Websites and Apps Now Fixed

 

[AndyK]

If you use 1Password you were never at risk anyway.

 

[SteveW928]

If you use 1Password you were never at risk anyway.

Pshew! That's good to know. But, I'd still rather manage the data myself (I use PasswordWallet by Selznick). But, if I weren't, 1Password would be my 2nd choice.

 

[Michaelgtrusa]

If you use 1Password you were never at risk anyway.

Glad to hear that.

 

[einsteinbqat]

Yay for 1P!

 

[Michaelgtrusa]

More here

 

[campyguy]

There's a couple of long lists of "possibly affected" sites, to be proactively updated, using Cloudflare posted on Github:
https://github.com/pirate/sites-using-cloudflare
Among them I saw authy.com, dlink.com, feedly.com, uber.com, 4chan.org, yelp.com, zendesk.com, and several of the Android-focused websites like Droid Life. It's a long list (several thousand long)…

And a short list of of "potentially affected" iOS apps here:
https://www.nowsecure.com/blog/2017/02/23/cloudflare-cloudbleed-bugs-impact-mobile-apps/
I saw CNN, BeInSport, FitBit, Western Union, Yelp, Outlook, and Dropbox on that shortlist.

 

[SteveW928]

There's a couple of long lists of "possibly affected" sites, to be proactively updated, using Cloudflare posted on Github:
https://github.com/pirate/sites-using-cloudflare
Among them I saw authy.com, dlink.com, feedly.com, uber.com, 4chan.org, yelp.com, zendesk.com, and several of the Android-focused websites like Droid Life. It's a long list (several thousand long)…

And a short list of of "potentially affected" iOS apps here:
https://www.nowsecure.com/blog/2017/02/23/cloudflare-cloudbleed-bugs-impact-mobile-apps/
I saw CNN, BeInSport, FitBit, Western Union, Yelp, Outlook, and Dropbox on that shortlist.

I guess what I'd want to see is a list of those where it actually matters. As shown by the 1Password example above, it didn't matter that they simply used Cloudflare.

 

[campyguy]

I guess what I'd want to see is a list of those where it actually matters. As shown by the 1Password example above, it didn't matter that they simply used Cloudflare.

I wish there was a list like that, I haven't seen one yet or I would have posted it. At least the Github list is annotated with sites that are "not affected", like 1Password. I tend to rotate my passwords regularly anyway and use 2FA, but I do appreciate the work that people do for the rest of us. Seeing the "change your passwords now" cry has been dulled a bit by reading this has been going on for several months now. Sigh.

Seeing other sites like PasteBin, whirlpool.net.au, and The Register on that Github list I'm planning to get to it soon, regardless. Time to break out the trusty old abacus and tin-can-with-a-string communication network. Cheers…

 

[CrystalQuest76]

Someone built a big database of the entities that use CloudFlare services. You can enter the url of your favorite service and see if its on the list.
http://www.doesitusecloudflare.com

I noticed Glassdoor is on the list.

 

[SteveW928]

I wish there was a list like that, I haven't seen one yet or I would have posted it. At least the Github list is annotated with sites that are "not affected", like 1Password. I tend to rotate my passwords regularly anyway and use 2FA, but I do appreciate the work that people do for the rest of us. Seeing the "change your passwords now" cry has been dulled a bit by reading this has been going on for several months now. Sigh.

I get how 1Password can say 'not affected' but I don't get how some of the others can. I have domains and DNS at Namecheap, and read their page about their investigation. I don't understand how they can say not affected, though. Unless I'm misunderstanding what happened, how would they even know (or be able to investigate)? No one logged into their site during the affected time periods?

Anyway, yea, using a password manager is a very good idea, as you can have a good, strong UNIQUE password for every site (i.e.: if one gets compromised, it's only that site). But, changing them can still be a pain for things like Dropbox, email, etc. where the change impacts all your systems and devices.

 

[Glassed Silver]

Someone built a big database of the entities that use CloudFlare services. You can enter the url of your favorite service and see if its on the list.
http://www.doesitusecloudflare.com

I noticed Glassdoor is on the list.

Cmd+F and a text list seems just as convenient.

What I'd like to see is Apple improve their password manager.

They do invest the effort to integrate an anti-virus system into macOS, yet when a service I use gets compromised my keychain is too "silly" to alert me that some of my credentials might be at risk.

And whilst you're at it, add a feature to manually trigger a keychain sync.

Oh and more iOS apps need to make use of keychain.

So far the only app that implemented this that I actually use is Amazon and good on them!

Glassed Silver:ios

 

[campyguy]

I get how 1Password can say 'not affected' but I don't get how some of the others can. I have domains and DNS at Namecheap, and read their page about their investigation. I don't understand how they can say not affected, though. Unless I'm misunderstanding what happened, how would they even know (or be able to investigate)? No one logged into their site during the affected time periods?

Anyway, yea, using a password manager is a very good idea, as you can have a good, strong UNIQUE password for every site (i.e.: if one gets compromised, it's only that site). But, changing them can still be a pain for things like Dropbox, email, etc. where the change impacts all your systems and devices.

Sorry for the delay, a client popped in with Scotch - here in the office we couldn't say no to either one…

I'd been poring over my own resources about getting to the bottom of this as well, and cruising the web for a more-narrowed or focused explanation and found one on Wired with some quotes from Cloudflare's CEO that broke it down for me, the whole post is a good read and the CEO's comments begin about halfway down starting with the "What Happens Now" header:
https://www.wired.com/2017/02/crazy-cloudflare-bug-jeopardized-millions-sites/

Keep in mind that the CEO cites a number of affected customers and not a number of affected web sites or portals… Cheers!

 

[Rigby]

I get how 1Password can say 'not affected' but I don't get how some of the others can. I have domains and DNS at Namecheap, and read their page about their investigation. I don't understand how they can say not affected, though. Unless I'm misunderstanding what happened, how would they even know (or be able to investigate)? No one logged into their site during the affected time periods?

Cloudflare offers a range of services (including content caching, DNS, application firewalls and DDoS protection). Not all of them deal with sensitive information (such as SSL/TLS traffic before encryption), and web sites don't necessarily use them for all data that they handle (e.g. some may choose to only use them for caching web site images or the like). So, depending on which Cloudflare services customers use and for what data, they may not be affected.

 

[SteveW928]

What I'd like to see is Apple improve their password manager.

No, no, no! Do NOT trust Apple with this kind of thing. First, Apple sucks at anything 'cloud' and 'sync' related. Second, they've already had one major vulnerability in regard to Keychain. Third, it's something they don't specialize at. For something this crucial, you want a 3rd party that specializes in encryption and security. I only recommend two options at the moment... PasswordWallet by Selznick, and 1Password.

Get one of the above. Create a strong, unique password for each service. And, answer those insanely insecure 'security questions' with random answers (also unique to the site you're using). Here's an article I wrote on the subject when the Yahoo debacle broke: http://www.cgwerks.com/yahoo-hack-password-management-problem-security-questions/

... found one on Wired with some quotes from Cloudflare's CEO that broke it down for me, the whole post is a good read and the CEO's comments begin about halfway down starting with the "What Happens Now" header:
https://www.wired.com/2017/02/crazy-cloudflare-bug-jeopardized-millions-sites/
Keep in mind that the CEO cites a number of affected customers and not a number of affected web sites or portals… Cheers!

Great find!

I especially like this gem:
"To mitigate whatever risk does remain, security researcher and former Cloudflare employee Ryan Lackey suggests changing every password for every online account, since the “Cloudbleed” leak could have exposed anything."

Cloudflare offers a range of services (including content caching, DNS, application firewalls and DDoS protection). Not all of them deal with sensitive information (such as SSL/TLS traffic before encryption), and web sites don't necessarily use them for all data that they handle (e.g. some may choose to only use them for caching web site images or the like). So, depending on which Cloudflare services customers use and for what data, they may not be affected.

For sure. What I meant, though, is that I'm betting Namecheap (and may other sites) are issuing the CDN aspect of their services. It's just a guess, of course, but if so, that would mean if someone logged into those sites, the perpetrator might have login info or cookies to be able to hijack an account. The reason 1Password is safe is that the info is layers removed and further encrypted. (if I'm understanding correctly) But, yes, I suppose it's possible Namecheap was using their services in another way... but that wasn't the impression I was getting. It sounded to me more like damage control... like, we don't yet know of any accounts being hijacked, etc. I hope I'm wrong on that, as I'm a Namecheap customer (and generally, quite a happy one).

 

[Rigby]

No, no, no! Do NOT trust Apple with this kind of thing. First, Apple sucks at anything 'cloud' and 'sync' related. Second, they've already had one major vulnerability in regard to Keychain. Third, it's something they don't specialize at. For something this crucial, you want a 3rd party that specializes in encryption and security.

Apple has some of the best and most well-respected security experts (I actually know a few of them), and they put far more resources into their security mechanisms than smaller companies.

I only recommend two options at the moment... PasswordWallet by Selznick, and 1Password.

Personally, I'd recommend an open-source solution (e.g. Keepass), since it is likely that there are more eyeballs looking for bugs and vulnerabilities. It is really not possible to judge the quality of closed-source apps, especially if they are not independently audited.

For sure. What I meant, though, is that I'm betting Namecheap (and may other sites) are issuing the CDN aspect of their services. It's just a guess, of course, but if so, that would mean if someone logged into those sites, the perpetrator might have login info or cookies to be able to hijack an account. The reason 1Password is safe is that the info is layers removed and further encrypted. (if I'm understanding correctly) But, yes, I suppose it's possible Namecheap was using their services in another way... but that wasn't the impression I was getting. It sounded to me more like damage control... like, we don't yet know of any accounts being hijacked, etc. I hope I'm wrong on that, as I'm a Namecheap customer (and generally, quite a happy one).

I don't know if that particular company is affected or not. I didn't mean to downplay the risks. This is potentially one of the most severe incidents yet, if only because of Cloudflare's large footprint. What I wanted to say is that in some cases it is possible for Cloudflare customers to determine with certainty whether or not they have been affected. But yes, there are many scenarios where this is difficult since the information disclosed was somewhat random (essentially cloudflare servers sprayed random memory contents when processing certain requests).

 

[SteveW928]

Apple has some of the best and most well-respected security experts (I actually know a few of them), and they put far more resources into their security mechanisms than smaller companies.

Maybe overall, especially the stuff in hardware on the phones, but since like Snow Leopard, most of the added features, especially cloud and sync related ones, are pretty poorly designed and implemented. I use some of them, of course, but don't trust most of them. (i.e.: Calendar, Contacts, etc.).

I don't trust Apple much in terms of software quality anymore, except that at least in principal, they seem to have better privacy/encryption intentions. But, the best of intentions, can be messed up by sloppy work.

Personally, I'd recommend an open-source solution (e.g. Keepass), since it is likely that there are more eyeballs looking for bugs and vulnerabilities. It is really not possible to judge the quality of closed-source apps, especially if they are not independently audited.

That might be a good option, too. The interface looks rather clunky, but I guess if it works, it works. While I'm a fairly big fan of open-source, I'm not sure the eyeballs/quality thing always plays out. It depends what the people on the project are focused on.
(ex: WordPress is how old now, and the comment system is total junk... and blogging is a core/use function. Or, on the security front, they just had a huge bug in the REST API implementation, which they defaulted to on - even though most sites won't use it - and didn't seem to take it very seriously. It's great software, but most of the devs' minds are elsewhere... from what you might value most.)

 

[Parasprite]

If you use 1Password you were never at risk anyway.

Note that this means your master password is safe if you happen to use a 1Password account. This doesn't mean that there is any less risk for other data (including logins and other sensitive data).

 

[Uffish Thought]

Note that this means your master password is safe if you happen to use a 1Password account. This doesn't mean that there is any less risk for other data (including logins and other sensitive data).

1Password has a security feature called Watchtower. It listed all my vulnerable logins and I changed those passwords, it took 10 minutes.

 

[SteveW928]

1Password has a security feature called Watchtower. It listed all my vulnerable logins and I changed those passwords, it took 10 minutes.

Yea, that's a good idea. I didn't change all mine, as having a strong, unique password for each means I don't care all that much if someone hacks into a low-priority account. But, I did change the big ones, like Dropbox, Namecheap, my main ISPs control system for my client's sites, and stuff like that. While it's rather unlikely something would have happened, it's far better to be safe than sorry (and as you said, it's not that hard).

 

[happywaiman]

Maybe overall, especially the stuff in hardware on the phones, but since like Snow Leopard, most of the added features, especially cloud and sync related ones, are pretty poorly designed and implemented. I use some of them, of course, but don't trust most of them. (i.e.: Calendar, Contacts, etc.).

While Apple can't design a decent cloud service that involves outside protocol (calendar, mail, etc), I don't think iCloud Keychain is that bad.
In all fairness, keychain corruption is still happening, and I see no plan from them to improve the quality of the service, but iCloud Keychain use the same pathway as iOS backup and Find My iPhone to push the upgrade, which is robust and stable (when have you got a problem in push? ), and it is actually implemented pretty good, especially considering they literally have one shot doing it. (All the secure key needed to fix any bug related to the access of the passwords is purposely destroyed by a blender to prevent any leak. I don't think there's a cloud service do it that way, or even use an HSM to safe guard against the brute force attack, since most app use public cloud as the backend)

Beside, as long as they don't drop any major public protocol in the cloud service, I would consider they are doing better than Gmail. I used to have real time push in Apple mail for Google using exchange in the SL era, before Google decided to make everyone reading habit one thing they track, or you got to find some organization to buy the G Suite for you

 

[gnasher729]

No, no, no! Do NOT trust Apple with this kind of thing. ..., they've already had one major vulnerability in regard to Keychain.

Citation needed.