It's been really interesting to watch the dramatic uptick in activity around the automation space the last year or two. I don't need to go into too much detail on the benefits that automation offers here; consistency and scalability are two of the more prominent that come to mind. What has struck me, though, is that it feels like the way that companies are going about it is missing a key step.
In the Enterprise we are used to hearing about maturity models. The Capability Maturity Model (CMM) for processes (originally software processes) being one of the most common. One of the key things you learn with maturity models though is that when progressing along them it is critical that you don't skip levels. That is the point of it being a "maturity" model. You grow through each stage, building on what you already have. You can move quickly through a stage but if you try to skip one you are destined for failure.
When looking at an IT automation implementation at a high level you can argue that it is simply a case of moving from manual deployment of systems to automated deployments. For a fuller picture though I believe that you need to take validation, or testing, into account. If I were to illustrate the maturity model, or roadmap if you will, for IT automation I would do it like this:
So why automate testing before automating your deployment? Think about it this way. What do you need to do before you start automating your systems? You need to gather your requirements for automation. Now you can gather them the classic way in a document of some sort but how much utility do you get from collecting them this way? Take a leaf from the book of software developers and test driven development and gather your requirements as tests instead. Why? There are multiple benefits:
1. If your requirements are executable you have test driven automation and you can track your progress with your automation implementation. How are we doing? Well how many tests are passing?
2. Your tests live on and become the safety net for your automations ongoing. Remember, automations are code too and they will have bugs. If you support them with separate tests then you can guarantee their quality.
3. You reduce vendor lock in for your automations. It is a much less risky task to switch from Puppet to Chef, for example, if you have an independent set of configuration tests to validate your migration.
So, not only is automated configuration testing a natural step on the road to full automation, it has multiple additional benefits.
How you automate your configuration testing is up to you. If you are comfortable with a particular scripting solution or testing framework then that is fine although portability and collaboration can suffer. Remember, configuration is something that multiple teams have a stake in, from development, operations, security and even testers themselves. If you are serious about bulletproofing your automations through testing you may want to consider a more fit for purpose configuration testing solution like.. um... oh yeah, UpGuard:)
However you implement though the mere exercise of doing so prepares you for automation. Whilst tempting to ignore it is always time well spent.