Breaches
Enterprise Scale: How Public Storage Buckets Leaked Private Credentials

Enterprise Scale: How Public Storage Buckets Leaked Private Credentials

UpGuard Team
UpGuard Team
Published Aug 24, 2020
Join 27,000+ cybersecurity newsletter subscribers

UpGuard can now report the discovery of multiple misconfigured cloud storage buckets under the control of Hortonworks, an enterprise data processing company which completed a merger with Cloudera in January of 2019. The massive amount of files accessible in those buckets were largely intended for public distribution, as Hortonworks contributes to and supports the open source Apache Hadoop project. Amidst terabytes of intentionally public files, however, were numerous system credentials and other internal developer information

Discovery and Notification

UpGuard analysts identified a cloud storage bucket configured for public access located at the URL "dev.hortonworks.com.s3.amazonaws.com" and proceeded to download and review a sample of the files. Initial analysis by the UpGuard team showed sufficient reason to believe sensitive information stemming from Hortonworks was most likely exposed to the public internet through the discovered file repositories. We initiated notification efforts on July 27, 2020.

2
Partial listing of the top level contents for one of the buckets, including directories marked for deletion like "BuildsToDelete."

On August 8th, UpGuard received an email from Cloudera stating that "investigation and remediation is complete", "[t]he S3 buckets will remain open for downloads", and that analysis had shown only three files potentially containing confidential information which had been removed by July 30th, 2020.

However, Cloudera's determinations would soon be revised. A follow-up email arrived the next morning. Cloudera had realized a backup of their company-internal collaboration and automation system– a local installation of the popular "Jenkins" software– was also available within the publicly accessible files. Jenkins is used to automate parts of the software development lifecycle, like building a software package and running tests. The backup of the configuration for Hortonworks' Jenkins installation stored here contained the usernames of developers and their encrypted passwords.

66_horton_employee_codereview_db_password_hashes
Redacted portion of username and encrypted passwords from Hortonworks' Jenkins backup.  

After receiving the follow-up email UpGuard analysts checked on the status of the bucket and found that all public access to the dev.hortonworks.com S3 bucket had now been removed.

Additionally, the email that morning from Cloudera's CIO informed our team that if we knew of additional relevant details, the information would be appreciated. As researchers who have reported many data exposures, we always appreciate a constructive response like this one. Open communication reduces risk and expedites remediation.

Chris Vickery, UpGuard's Director of Risk Research, replied with a list of 8 additional buckets we knew of bearing either "hortonworks" or "cloudera" in their title, three of which we knew were still publicly accessible at that point in time but did not appear to contain files at the same level of security concern as the original discovery.

Significance

While the vast majority of files we reviewed were indeed the kind normally made available for public consumption and community contribution, this incident illustrates the risks inherent in extremely large cloud storage containers. After several hours of recording the names of files available for download, the resulting list was 2.4 gigabytes of pure text– and that was just the names of the files with no content. The scale of these storage containers makes it such that even fully automated processes (like downloading and text searches) were slow by human standards, to say nothing of the time it would take for a human to review the contents. But within that vast collection were files that contained credentials to systems that appear to be core to Hortonworks' software development.  

1

This single file, named hwx_secrets.conf, contains plaintext credentials for numerous other systems. UpGuard never attempts to use discovered credentials, so the potential impact of this exposure is unknown, but their potential reach extends into multiple sites for the production and delivery of software. There are passwords for:

  • The user "relengjenkins" on Linux
  • The user "jenkins-daemon" on Windows
  • Three API keys for authentication on Github
  • A database password
  • A Jenkins signing key
  • Credentials for users on the public and private Nexus instances
  • The "releng" Jira instance

When that many directories and files of varying format are all stashed away together, it becomes all too easy for something to be mistakenly put among them and remain unnoticed, as is what appears to have happened here. Even after their designated security response email address received UpGuard's formal notification of a potentially serious security concern, it took eleven days for Cloudera/Hortonworks to recognize the true gravity and scope of concern.

Conclusion

Hadoop, Hortonworks’ primary product, is for big data problems at the largest scale. Facebook, for example, uses Hadoop; Cloudera’s home page currently boasts that their customer base includes most of the top ten largest companies in any industry. And both Cloudera and Hortonworks company names are present among the portfolio of In-Q-Tel, the private investment branch of the CIA. For companies that develop technology central to the backbone of huge swathes of commerce and network resilience, security practices preventing data leaks and accelerating incident response are vital.

UpGuard customer support teamUpGuard customer support teamUpGuard customer support team

Protect your organization

Get in touch or book a free demo.
Abstract shapeAbstract shape

Related breaches

Learn more about the latest issues in cybersecurity.
Deliver icon

Sign up to our newsletter

Get the latest curated cybersecurity news, breaches, events and updates in your inbox every week.
Abstract shapeAbstract shape
Free instant security score

How secure is your organization?

Request a free cybersecurity report to discover key risks on your website, email, network, and brand.
Website Security scan resultsWebsite Security scan ratingAbstract shape