Managing change in a busy, often chaotic technology environment is a historically difficult discipline even for experienced IT professionals. Finding the balance between risk and reward inherent to all changes takes skill, experience, and patience. Developing a change management process that protects your infrastructure, without throttling the pace of innovation is a skill that can take an entire career to master.
System engineers and administrators typically loathe change management. They view this aspect of IT Service Management as a necessary evil, an obstruction that serves only to slow down the rate at which work is completed. Not to mention this is before automation is introduced to the environment.
Automation for the on-prem crowd is in a state of awkward adolescence. We’ve thrown the term around long enough; everyone agrees that automation should play a central role in all IT shops, from large multi-nationals to small single-office entities. The fact that every new solution has an API available out of the box underscores how pervasive automation has become. It’s no longer a feature, it’s an expectation.
IT professionals embrace automation because it solves a practical problem: how do you build a repeatable process and manage infrastructure at scale. In other words, how can you turn configurations into templates, and deploy from those templates on demand?
But what happens when automation runs head-first into change management? Are these two forces in conflict?
Controlling Automation
At first, controlled automation may seem like an oxymoron. If you force automation through the same rigorous change management practices that your manual, labor-intensive tasks follow, are you defeating the purpose? For tired, over-worked operations staff, automation is a miracle. Can you schedule a miracle?
A great example of an early clash between automation and change management is virtual machine migration. In first generation virtualization deployments, VMs were generally bound to a single hypervisor. Migrations were performed infrequently, and it was not uncommon to maintain a spreadsheet that listed each host and its guest VMs. Documentation of your environment was a relatively static exercise. But with technologies that made live migrations possible, and then the introduction of scheduled resource distribution, the ability to track host-to-guest relationships became futile; by the time you’d made the spreadsheet, it was incorrect.
Change management types, with their rigid constraints on what constitutes a change, viewed live migrations of VMs as unapproved changes at first. The rate of technological innovation had outpaced the evolution of change management.
Eventually, as virtual machine migration technologies became commonplace and reliable, change management practitioners relented and acknowledged that it was less important to record every single VM migration, and more important to understand the automated and repeatable function of live migration itself.
When we shift from VM automation to network automation, we can expect the same growing pains within the change management organization.
ZTP vs. Bureaucracy?
As basic networking tasks become automated, and as that automation becomes as routine as VM migrations, network engineers will undoubtedly find themselves locked into conflict with the ITIL crowd. Once again, the debate will be over automated change: how can you provision networking, configure VLAN access, and do so on-the-fly without going through change control? Aren’t these all changes?
Consider Cumulus Networks’ Zero Touch Provisioning (ZTP). With ZTP, a savvy network engineer can deploy an entire switch from an image on a USB drive. And that image can include instructions to have the newly deployed switches listen to new Nutanix AHV nodes in order to automatically provision a network to support not only that node but its VMs too. We’re talking about a lot of changes being made, in real-time, by switches and hosts working in tandem.
Recall that the goal of change management is not to slow down the rate of innovation, or to create unassailable mountains of change requests to satisfy the desires of rabid auditors. The goal of change management is to protect the IT environment from disruption as the result of poorly planned or uncoordinated changes. With this frame in mind, you can present network automation as solution to one of change management’s oldest problems: how do you ensure that routine changes are made the same way every time?
When a config file dictates how MLAGs are provisioned, or how VLANs are assigned, or how logging levels are defined across a device, you end up with not just automation that saves time, but repeatable procedures that never make mistakes. There’s no fat-fingering the access group, no copy/pasting the wrong VLAN on the wrong port. When your server team is ready to cable up a few new Nutanix blocks, you can depend on the accuracy of the network config because it’s all pre-determined and automated.
Without the Cumulus Networks and Nutanix partnership, these provisioning tasks would be manual and error-prone. So how do you find common ground with your change manager?
There’s no conflict to address here when you keep this in mind: network automation directly addresses the change manager’s need to ensure the availability of the infrastructure and its applications. When network automation tools like ZTP, and the seamless integration between Cumulus OS-based switches and Nutanix blocks, your network provisioning and day 2 operations become repeatable, routine, and reliable. That will brighten any bureaucrat’s day.
Great article but it leaves me asking “show me more.” Maybe hit the point home with some some example tooling, practices, and process enhancements. The idea of “repeatable procedures that never make mistakes” sounds promising but I often see well-intended changes get pushed through the trusted automation pipeline but cause catastrophic issues. This makes the change auditors even more concerned about the automated process.