Updated on November 28, 2017 by Dan O'Sullivan
UpGuard’s Cyber Risk Team can now report that a misconfigured cloud-based file repository exposed the names, addresses, account details, and account personal identification numbers (PINs) of as many as 14 million US customers of telecommunications carrier Verizon, per analysis of the average number of accounts exposed per day in the sample that was downloaded. The cloud server was owned and operated by telephonic software and data firm NICE Systems, a third-party vendor for Verizon.
The UpGuard Cyber Risk Team is a unit devoted to discovering data exposures where they exist, helping to secure them and raising awareness about the issues of cyber risk driving data insecurity across the digital landscape.
The data repository, an Amazon Web Services S3 bucket administered by a NICE Systems engineer based at their Ra’anana, Israel headquarters, appears to have been created to log customer call data for unknown purposes; Verizon, the nation’s largest wireless carrier, uses NICE Systems technology in its back-office and call center operations. In addition, French-language text files stored in the server show internal data from Paris-based telecommunications corporation Orange S.A.—another NICE Systems partner that services customers across Europe and Africa.
Beyond the risks of exposed names, addresses, and account information being made accessible via the S3 bucket’s URL, the exposure of Verizon account PIN codes used to verify customers, listed alongside their associated phone numbers, is particularly concerning. Possession of these account PIN codes could allow scammers to successfully pose as customers in calls to Verizon, enabling them to gain access to accounts—an especially threatening prospect, given the increasing reliance upon mobile communications for purposes of two-factor authentication.
Finally, this exposure is a potent example of the risks of third-party vendors handling sensitive data. The long duration of time between the initial June 13th notification to Verizon by UpGuard of this data exposure, and the ultimate closure of the breach on June 22nd, is troubling. Third-party vendor risk is business risk; sharing access to sensitive business data does not offload this risk, but merely extends it to the contracted partner, enabling cloud leaks to stretch across several continents and involve multiple enterprises.
NICE Systems’ history of supplying technology for use in intrusive, state-sponsored surveillance is an unsettling indicator of the severity of this breach of privacy. This offshore logging of Verizon customer information in a downloadable repository should be alarming to all consumers who entrust their private data to major US companies, only to see it shared with unknown parties.
On June 8th, 2017, UpGuard Director of Cyber Risk Research Chris Vickery discovered a cloud-based Amazon S3 data repository that was fully downloadable and configured to allow public access. The database and its many terabytes of contents could thus be accessed simply by entering the S3 URL.
The repository’s subdomain, “verizon-sftp,” is an indication of the files’ corporate origins. Viewing the repository, there are six folders titled “Jan-2017” through “June-2017,” as well as a number of files formatted with .zip, among them “VoiceSessionFiltered.zip” and “WebMobileContainment.zip.” These files, inaccessible via .zip extraction, could be decompressed once the format was changed to .gzip, another file compression program.
The “verizon-sftp” repository.
Each month-named folder contains directories corresponding to each day of the month. Within each of these day folders are a couple dozen or so compressed files. By every indication, this is a repository for the automated daily logging of files. The folder for “June-2017” records a halt to logging on June 22nd.
The daily log folders in the “Apr-2017” folder.
Once unzipped, the contents of these daily logging folders are revealed to be sizable text files, some as large as 23 GB. Analyzing them, the general structure becomes apparent: the large text blocks appear to be composed of voice recognition log files, the records of an individual’s call to a customer support line, including fields like “TimeInQueue” and “TransferToAgent.” Pings to various subdomains of https://voiceportalfh.verizon.com further indicate the voice-activated technology producing this data.
This is not all, however. A great many Verizon account details are also included in the logs, such as customer names, addresses, and phone numbers, as well as information fields indicating customer satisfaction tracking, such as “FrustrationLevel,” and service purchases, such as “HasFiosPendingOrders.” Values including number ratings, “True,” “False,” “Y,” and “N” are assigned to each field. For a large amount of these logged calls, however, the most sensitive data—such as “PIN” and “CustCode”—is masked.
A call log, with the most sensitive data masked.
But not all of the records have these details “masked” in this manner. For a smaller amount of these logged calls, there is no such masking at all—revealing such details as unmasked “PIN” codes. Such account PINs are a crucial part of verifying callers as legitimate customers, ensuring impersonators cannot access and change Verizon account settings. Other fields and their answers, such as “CallCenterPassword,” indicate which account-holders have requested a higher standard of security for customer service calls to change account settings, allowing any potential scammers in possession of the logs to determine which customers would be easier to victimize. In one such text file, there were six thousand such unmasked PIN codes.
A call log, with the most sensitive data exposed (here redacted by UpGuard).
Less immediately explicable is the presence in the S3 server of data originating from French telecoms provider Orange, another partner of Nice Systems and one with which Verizon competes in the European data market.
French-language data originating from Paris-based telecom Orange S.A.
While it appears this internal Orange data is less sensitive, it is noteworthy to see such information included in a repository otherwise devoted to Verizon.
The critical data repository in question was exposed not by the enterprise holding primary responsibility for the information, but by a third-party vendor to the enterprise. It was a publicly accessible AWS S3 bucket owned by third-party vendor NICE Systems that revealed the sensitive personal details of Verizon customers.
To judge by much of its website copy and marketing material, NICE Systems is indeed a company that provides technology of particular use to call centers, a crucial component of the Verizon business chain. SEC filings reveal NICE Systems to call Verizon a “main partner,” providing the telecom carrier with such software as a workforce management tracker to monitor how efficiently call center operators are using their time. Other programs offered within the suite of NICE Enterprise software include data and voice analytics software, technology in which NICE has made significant investments as crucial to call center customers.
Beyond such direct business, a series of high-profile US acquisitions by the Israeli firm have given them an even closer business relationship with Verizon’s North America operations than might be immediately apparent. In 2016, NICE acquired inContact and VPI, both firms that have in the past supplied Verizon with software for its back-office and call center operations.
In short, NICE Systems is a trusted Verizon partner, but one that few Americans may realize has any access to their data. Such third-party vendors are entrusted every day with the sensitive personal information of consumers unaware of these arrangements. There is no difference between cyber risk for an enterprise and cyber risk for a third-party vendor of that enterprise. Any breaches of data on the vendor’s side will affect customers as badly and cost the business stakeholders as dearly as if it had been leaked by the enterprise.
Beyond the sensitive details of customer names, addresses, and phone numbers—all of use to scammers and direct marketers—the prospect of such information being used in combination with internal Verizon account PINs to takeover customer accounts is hardly implausible. To do so would enable impersonators to tell Verizon call center operators to do whatever was wished of them—enabling, perhaps, costly “SIM Swap” scams of customer SIM cards, or, as reported by The Verge, the breaching of two-factor authentication:
“Two-factor’s trickiest weak point? Wireless carriers. If you can compromise the AT&T, Verizon, or T-Mobile account that supports a person’s phone number, you can usually hijack any call or text that’s sent to them. For mobile apps like Signal, which are tied entirely to a given phone number, it can be enough to hijack the entire account. At the same time, carriers have been among the slowest to adopt two-factor, with most preferring easily bypassed PINs or even flimsier security questions. With two networks controlling the bulk of the market, there’s been little incentive to compete on security.”
The prospect of a host of your applications and digital accounts being compromised from one third-party vendor’s exposure of data is not science fiction, but the unfortunate reality of cyber risk today. The data exposed in the Verizon/NICE Systems cloud leak is, indeed, a testament to how profoundly every aspect of life today is touched by those systems to which we impart so much knowledge.
Misconfigurations are an internal problem that emanate from within the IT infrastructure of any enterprise; no hacker is necessary for massive damage to occur to digital systems and stored data. And the problem is pervasive, with Gartner estimating anywhere from 70% to 99% of data breaches result not from external, concerted attacks, but from internal misconfiguration of the affected IT systems.
Join the newsletter to receive UpGuard Breach Analysis post alerts via email
Cyber Risk Analyst
As a security researcher, Chris possesses a long track record of professional distinction and success discovering major data breaches and vulnerabilities across the cyber landscape. Previous achievements have included reports on the online exposure of hundreds of thousands of patient medical records, an illegal spam operation insecurely storing 1.4 billion target email addresses, a series of unsecured MongoDB databases which threatened the security of millions of internet users, and a publicly accessible database containing the voter registration records for 93.4 million Mexican citizens. As a researcher and analyst, Chris has helped to protect the most sensitive data of hundreds of millions of people, while raising awareness of the looming issues of cyber risk.
Cyber Resilience Analyst
Dan’s writing and journalism has appeared in VICE, Rolling Stone, Salon, The Daily Beast, Gawker, and Deadspin. His essay for Jacobin about labor exploitation in professional wrestling was anthologized in The Best American Sports Writing 2015. As a writer and analyst, Dan communicates the most important and relevant lessons of data breaches to both technically-proficient readers and to a general audience of those most affected by cyber risk.