If I say the word “patch” to you, you either think of a pirate like my six-year-old or you panic and ask yourself if you’ve applied the latest software updates to your laptop or the systems that you manage. Both inspire terror in the hearts of their audience, but only one is the subject of bad jokes. The other is a bad joke that we’ve turned into a career.
Peg-Leg Patching
Patches are probably both the easiest way to fix problems and the last resort to solving issues. Why is that? Why do we start shying away from installing updates as soon as the suggestion comes up during a support call or routine maintenance?
The problem is similar to a pirate’s peg leg. A wooden peg grafted onto your ankle or knee is a necessity to restore mobility to a pirate that doesn’t have a foot. It’s not a perfect solution but it’s a good-enough patch to keep a pirate upright. But a peg leg creates a bunch of new problems after installation:
- Do I need to make a shoe for it?
- What about stepping in knot holes? Or sand?
- Will beavers be an issue for me?
- What is the performance difference of an oak peg leg versus an unpatched leg?
As you can see, there are a ton of issues brought up just from one simple installation. Software is no different. When you are trying to figure out if a patch is going to solve a problem, you have to take into account how severe the problem is and how much the patch is going to cause issues with other components of the system.
Patch, Patch, Patch, and Patch Some More
If you tuned in to the recent Security Field Day event in Silicon Valley, you might have seen the great presentation from Bob Plankers (@Plankers) at VMware. He did a great job of running down all the security implications of virtual systems. Here’s one of those great talks about CPU vulnerabilities:
What I love, aside from the fact that Bob’s answer to every security question is “patch, patch, patch” is that he does a great job of understanding the trade offs with patches. For example, Spectre and Meltdown were huge security risks. Using CPU branch prediction to steal data is a big problem because it happens at a level that is undetectable by the host OS. So just patch right? Except patching those particular vulnerabilities also reduces CPU performance by 20% or more. That’s a huge hit to an individual system. What if that system is running dozens of VMs?
Patching has to have a cost benefit analysis. Let’s go back to our friendly pirate ship. The main sail has a hole torn in it. The captain is now dead in the water. They have enough material on board to patch the sail but it means they have to go slower and risk getting caught by the Royal Navy. It also means sacrificing some of their booty. Should they patch the sail? Well, the answer in this case is a pretty resounding “yes” since being dead in the water is not an acceptable solution. There’s risk involved with the choice and it’s not cost-free, but it’s the best option.
Likewise, when examining how to patch your own systems, you have to take into account all the factors. You have to know the impact of reducing hardware performance or introducing unstable code into the system. If the vulnerability is minor or not likely to be widely exploited, the best solution may indeed be to not patch. Bob might not like you in that case but the costs outweigh the benefits. In most security-related scenarios the costs of not patching are far greater than any supposed benefits of keeping the system static.
Raiding the DevOps Privateers
One last note on patching would be that it’s critical but distinct. This one is for all the DevOps folks out there that claim rapid development and deployment are other solutions to the patching problem. If you neglect the patches on your existing software, you’re going to be building on a pile of quicksand.
Think about game development. There are always teams working on pushing out new features to the game. Adding maps. Creating new features. Adding content to keep players engaged. But there are also people dedicated to fixing the broken stuff. Some of that broken stuff is super critical, like exploits or performance issues that knock the game offline. Other patches aren’t as critical, such as fixing textures or resolving minor non-game-breaking bugs that have been around for years. But patches happen separate from other development. Just because you’re pushing new features doesn’t mean you can neglect patching. You need both.
Bringing It All Together
Ultimately, patching is the right answer. They fix all your ills, mostly. But you also need to patch intelligently. You need to know which patches need to be pushed silently in the middle of the night to resolve zero-day security issues and which ones can wait until the next scheduled maintenance window. And you need to understand the benefits of each patch and be ready to deal with exceptions when they happen. Otherwise you’re going to find yourself walking the plank when management comes asking why the systems are down.
