UpGuard made detecting and fixing the Heartbleed vulnerability a lot less painful than it could have been. With a combination of Search and Policies we quickly and easily gained confidence that our servers were patched.
Like many people, the first information we found out about the Heartbleed problem was from the Heartbleed.com website. It told us we should upgrade to OpenSSL 1.0.1g immediately. Over to UpGuard to see if we're affected.
UpGuard scans all our machines nightly which allows us to easily search across our fleet.
We like to keep on top of all the latest security updates and sure enough we already had the latest version available, but it doesn't have a "g" after it. What's going on? A quick view of the version and changelog on one of our servers gives me some more information:
$ openssl version -a OpenSSL 1.0.1 14 Mar 2012 built on: Mon Apr 7 20:33:29 UTC 2014 ...
Ok, so the package was updated recently. Good. What changed?
$ apt-get changelog openssl openssl (1.0.1-4ubuntu5.12) precise-security; urgency=medium * SECURITY UPDATE: side-channel attack on Montgomery ladder implementation - debian/patches/CVE-2014-0076.patch: add and use constant time swap in crypto/bn/bn.h, crypto/bn/bn_lib.c, crypto/ec/ec2_mult.c, util/libeay.num. - CVE-2014-0076 * SECURITY UPDATE: memory disclosure in TLS heartbeat extension - debian/patches/CVE-2014-0160.patch: use correct lengths in ssl/d1_both.c, ssl/t1_lib.c. - CVE-2014-0160
Great. Ubuntu patched the vulnerability in their package without updating the OpenSSL version number. So we've got the right version.
What if we don't have the right version, and how can we be sure we're protect going forward? Here is where a UpGuard Policy helps.
Policies contain tests that can be scheduled to run regularly against a node or a group of nodes. I'm going to create a Policy package test to make sure all my nodes have been patched. I could even get UpGuard to install the latest version for me:
Now I can be sure I'm covered and that any machine I bring up in the future will also be covered.
Please note that this has only covered patching the OpenSSL package. Any running services will need to be restarted, otherwise they may be using the old, vulnerable version. You may also need to regenerate all server SSL keys and any other sensitive information as required.