Apple's $20M Configuration Problem

Updated on February 26, 2016 by Jon Hendren

This week, Apple’s App Store and iTunes Store suffered a downtime of about 10 hours. For the better part of the day, customers were unable to access the stores, purchase music or apps, or make payments using the Apple Pay payment system. The problem has been attributed to “a configuration blunder” of its DNS setup.

Apple—the most valuable company in the world, which at one point held more cash reserves than the United States government—sells approximately 15,000 songs per minute through iTunes according to 2013 figures, and facilitates $10 billion in consumer spending on apps via its iconic App Store. That in mind, we can do some middle-school-level math to ballpark lost revenue at approximately $9M for music and $11M+ for apps. The Register’s ballpark figure of $21M isn’t far off, either.

The company’s been characteristically tightlipped about the exact cause of the problem or why it took so long to remediate, but we couldn’t help noticing that finding configuration discrepancies is exactly what we do. Like, to the letter.

UpGuard scans and monitors the configuration state of practically any device in your IT environment, whether it’s an OSX, *nix, or Windows server, a router or switch, cloud apps, or really, whatever you’ve got. If it’s got a configuration state and you can log into it, chances are good that UpGuard supports it out of the box.

And by “configuration state,” we mean everything about how a device is configured: Installed software, user accounts, ports open, services, as well as application config files you want monitored. So if, y’know, some DNS change got made somewhere, you’d know exactly where it happened and where to go to fix it.

Once your configurations are scanned into UpGuard, you can compare them not only to previous scans, but you can compare machines to each other or to groups of machines, ensuring that your configurations are consistent across your entire environment. And when a deviation is detected, it’s automatically made into a task which can then be assigned to a team member to either remediate or sign off on, giving instant visibility and accountability into every change that occurs.

UpGuard group diff
We’ve also designed the interface to not only be unique, but to be easy enough for everyone on the team to use. No longer is visibility gained through the Ops guy’s incomprehensible shell scripts. Now every stakeholder up the chain can log in and see at a glance how the entire environment is doing.

See your website's faults before your competitors

Configuration drift has been a fact of life in IT for as long as IT has been a paying gig. UpGuard keeps it from spiraling out of control. Log in and give it a shot for up to 5 nodes for free, or if you’d like us to give your team a full live demo, we’d be more than happy.