Cloudflare Reverse Proxy Bug Leaked Uninitalised Memory
-
It turned out that in some unusual circumstances, which I’ll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.
For the avoidance of doubt, Cloudflare customer SSL private keys were not leaked. Cloudflare has always terminated SSL connections through an isolated instance of NGINX that was not affected by this bug.Google cached pages found that they had crawled "private messages from well-known services, PII from major sites that use cloudflare, and even plaintext API requests from a popular password manager that were sent over https".
They seem to have responded pretty quickly to Google's Project Zero team
cloudflare quickly reproduced the problem, told me they had convened an incident and had an initial mitigation in place within an hour.
"You definitely got the right people. We have killed the affected services" -
Cool, too bad that they found an issue, but awesome how quickly it was fixed.
-
We just got this in email...
Dear Cloudflare Customer:
Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare's systems. If you haven't yet, I encourage you to read that post on the bug:
https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers' sensitive information could still be available through third party caches, such as the Google search cache.
Over the last week, we've worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.
In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.
Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.
To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.
Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.
Matthew Prince
Cloudflare, Inc.
Co-founder and CEO -
Awesome transparency.
This type of transparency is why I liked LastPass, too.Still worried about LastPass under its new owners.
-
@JaredBusch said in Cloudflare Reverse Proxy Bug Leaked Uninitalised Memory:
Awesome transparency.
Yeah, very happy about this.
-
Wordfence blog mentions abou this and shared some info and what to do for the sites thats connected with cloudflare. Wordpress for example its adviced to change the salts.
https://www.wordfence.com/blog/2017/02/cloudflare-data-leak/
-
For changing salts https://api.wordpress.org/secret-key/1.1/salt/
-
Its even worse than cloudflare describes it, affects including 2FA
-
Full list of sites using cloudflare includes medium, digitalocean etc https://github.com/pirate/sites-using-cloudflare
1Password confirmed that it is not affected https://discussions.agilebits.com/discussion/comment/356869/#Comment_356869
-
I am wondering...if you use CloudFlare for public DNS record hosting only and chose not to accelerate any of your records across their CDN, would you be at risk based on what was discovered in their "leak?"
-
@NetworkNerd said in Cloudflare Reverse Proxy Bug Leaked Uninitalised Memory:
I am wondering...if you use CloudFlare for public DNS record hosting only and chose not to accelerate any of your records across their CDN, would you be at risk based on what was discovered in their "leak?"
No
