It was a nice summer morning, I was sitting on my back patio relaxing with a coffee watching the chickens, dogs, and other animals then this happened.

I agree and wanted to spend a moment reflecting on various points in my career, my own actions, and my opinions. I will be doing a few posts when I have time discussing various points in my career where I have faced challenges and knew what to do and it was not enough. My thinking was not evolved alone, it was the diverse colleagues around me who dove into these problems with me, provided perspective, and ideas and shaped my opinions to what they are today. I can not take that credit, but I also have to many people to list them one by one.

Just patch

Just patch I said, its easy. In more words than that I told 3000+ people to patch OpenSSL (pre-Heartbleed 2011) on all their services. I was new to the role as a junior engineer at the company and I lost usable access to my email for a period. When I finally got access to my email, it was a lot of wrong people trying to inform me on how wrong I was. Pfft. Software Engineers.

I read every email, I answered most. I showed "good" ownership, however I listened to few. The ones I listened to where confirming my bias, that this was righteous. The ones I ignored had details I did not care about, implementation at a service level. I had loads of emails telling me how hard it was and why. I knew I had a mandate and it did not matter we all had to do it.

I continued to get better at measuring the problem and burn down, I had some scripts running to automate the accountability. When dates came to pass that I communicated, I went from email notification to ticket notification. I talked about burn down rates, I reported on it monthly to senior leaders, I thought I had made a difference and I was efficiently automating away my pain.

Thinking this was success I said if we continue to do this, they will continue to do the work! This started a path of multiple years trying to change things by just getting people to do what we knew they needed to do. This is not change but it might be abusive.

Senior Security Engineers scoffed said it would not work at scale, I said it did. In hindsight, we both nailed it, they were right. They wanted change, I wanted immediate impact and thought that would bring change. This behavior is not unique to me, I see it in my various colleagues and social circles today. We believe that the righteous must be done and anyone on the other side "doesn't get it" and "it's a waste of time".

What did I know?

I know that patching is one of the quickest ways to reduce security risk. You can out patch some of your adversaries, it is a war of resources.

What change did I want?

I want every adversary to be forced to operate at their highest and most expensive capability every single time. This means no known unmitigated vulnerability in the environment. At the time that was not what I wanted, but it is how I would phrase it today. What I wanted then was people to just do their job as "owners" of the service and provide it the minimal feeding and care it needs. I wanted people to perform what I would call basic hygiene.

Did I achieve it?

No. What I achieved was a stick, a big stick one that I used at times to alienate the very people I needed to work with. No goal was accomplished but massive amounts of good work went into it.

What did I learn?

At first, I was mentally lazy and chalked it up to others being lazy and proof that it had happened once. I continued notifying people but I started to build some of my own tooling so I could manage some of this better. Then I started notifying myself of issues in my services. I knew at this point it was just lazy people and I will go show myself we can do this easily because duh, it is just patching. I started debugging random things that where not part of my service, why isn't this random library that is a dependency of a dependency not compiling in my environment? I struggled, I walked away from patching my own services and started to try to build in a place that would not require it. Less dependencies, go to managed runtimes, anything I could do to get rid of the patching. I recorded that not as patching is hard, but I am not a software developer here and this build stack does not make sense. I was wrong, patching is hard.

Fast forward many years, patching still a problem but one I had stopped focusing on. During which I had went from engineer to manager and we had built some mechanisms to make sure we were patching our systems. We took time to do so, we included it in sprints, it slowed us down, but we knew it was important. We adapted to the pain. A group of engineers and I started looking at the problem from the lens of we have spent a lot of time and effort, we can see the result we have achieved, and we do not like it. What do we need to do differently?

We talked with software developers, we looked at the various asks people had had, we looked at our guidance, I came back to people are lazy, and the business does not prioritize patching.

First let us go after "people are lazy'. Here is my stance, people fundamentally want to spend the least effort for the most reward. If we look at how these service owners get rewarded its via new features increasing cash in and efficiency of lowering cost for cash out. If patching is high effort (it was) and does not directly increase cash flow (it doesn't) and your argument is based on something might happen, what chance do you have? So, while as a service owner I want to patch its just not economically viable for me to do all the time because the business does not prioritize it.

When I say the business does not prioritize it, I do not mean they do not want it, they want it in addition to the existing workload. They want it as free as it can be, they want it efficient. I believe it is fair for the business to prioritize providing value to its customers as well.

I was taught focus on the outcomes, but I became blind to them and ignored the blockers. I blamed others for my lack of vision into the blockers. I did not believe the blockers where my problem only the outcome and others needed to solve the blockers. I did not dive into the problems, I avoided them. I was part of the problem.

What would I do different now?

I think it is unfair to leave this here, there is a path forward and I do not believe in defeatism. I believe there are those that bend reality and those that live in it. So, in short, get out of the way.

This is my current thought process on the path, I will be refining this as I learn.

  • Focus on the mechanism of patching. How do people do it, what are the pain points, how often do they do it, how often does it fail. If they do not do it why not?
  • Find areas where you may be able to build automatic patching mechanisms. Do you have a build pipeline? Are you 100% cloud and using auto scaling groups? How are you deploying applications?
  • Partner with stakeholders of the various services and mechanisms and solve the problem of why people do not patch consistently. Service owners want to be good service owners. Help them solve the problem they face.
  • Focus your engineering talent on building minimal cost patching mechanisms. Can you make it free to the service owner? Make it so simple and so low cost that it is a why would I not, not why should I.

I knew what to do, I did not know how to do it.