The Three Phases to DevOps in Security (The Real Magic) - PART III Change Processes
This is part three in a five-part series. If you’d like to understand more of the stories behind this work and how it can be applied to making your security organization more effective, collaborative, and engaged, contact us….
1. Allocate Expertise: I take stock of the team, their expertise, strengths, and, of course, challenges. I also ask what their interests and goals are – what do they like doing, what do they want to do? With this information I divvy up responsibilities across the security team. While the structure depends on needs of the organization and available skills, I make sure I’ve created comprehensive coverage of responsibilities. I then collectively let the team know what my thoughts are, let them challenge them and point out what I might have missed, what things need to be added and where someone feels strengths are not being leveraged properly. It is ultimately about recognizing expertise in the team and making sure that expertise is externalized so that everyone knows who they can turn to.
Result: Recognition of expertise in your team, and point to them as the go-to for answers
2. Embed in Projects: Now to break down more walls. This is how I ensure that the team not only learns about what is going on in the organization, but also participates in creating the solution. I assign the more experienced security people in my teams (those with broad insight) to projects within the company. If the effort has a significant need for security, they become the security Program Manager – the person who triages all requests from security and who acts as liaison between the project personnel and security specialists within the security team. This Program Manager needs to be very involved – participating in as many project meetings as possible, engaging with the project personnel, regularly communicating needs and “Managing by Walking Around”.
I’ve made this arrangement at every client I’ve led. I’ve had some people take to this like a fish in water – they love the interaction and actively participate with the project team, feel part of the team, and take its success personally. In one particular case we attended a new project initiation. We listened, recognized that security was not a material consideration for the project, provided non-security feedback and questions, and were rewarded with a big thank you. They appreciated our input and made a habit of inviting us for every new project they considered.
Result: Engaging and Embedding in projects. Knowing what happens in the organization. Create low friction, high return work environments where security is perceived as being invested in the success of the project – through the time committed and the willingness to listen and care about the goal of the project.
3. For Every Control You Implement You Must Give Something Back: This one probably sounds like process, but at its heart is empathy. Security teams have a tendency to impose controls that make tasks harder or take longer. This is a problem for those people in the company trying to get their work done in time for a deadline imposed by their manager. In an effort to meet the deadline they will be willing to take any steps to achieve that goal – including side-stepping security. Security needs to empathize with this. Hence my rule.
I ask my security teams to look at what their requirements are taking away in terms of efficiency. How much time is lost because people have to go through the security control (perhaps versus the old way they did things, or versus the intended design). I’ve had discussions about complex passwords being a requirement, and I ask about the challenge of remembering complex passwords, versus something less cumbersome to remember – like a biometric attribute (a finger) that is rather hard to forget, or multi-factor authentication where a token is something to lose, but using a mobile device is less likely to be forgotten because it is ingrained in our culture and daily lives. Even elements such as change control, or segregation of duties can be examined carefully to identify the real objective behind the control, and how can it be arrived at in a manner that is far less cumbersome and obtrusive yet deliver the same level of resistance to a threat.
Result: A mentality around the potential effects of security, and a thoughtful approach that looks to minimize that impact. A view that security actually cares and is sensitive to personal success.
4. Prioritize – Be Great at Important Things: This is where I insert a bit of security – but where understanding the customer comes strongly into play. Nearly every organization struggles with how to prioritize its work. For Security Teams, this includes the endless list of “must have” projects, tickets, vulnerabilities and audits. How do you decide what goes first? Some use a thumb in the wind, while others claim “professional expertise”. I prefer measures and collaboration in the team. I force the team through what I call “Risk Week”. It’s a week-long session (that gets shorter over time as they get great at it) where we create our risk model and mitigation priorities for the year. It is a highly collaborative effort. It includes revisiting all the organizational groups. It includes assigned responsibilities within the team so they all participate. It involves learning how to measure and make the exercise objective and repeatable. It involves presenting their ideas, each participant challenging assumptions, and creating active discourse as priorities are weighed. We even include an executive presentation where the team is welcome to present so that they gain the experience and the exposure.
Result: Risk Assessment that is based on the company’s goals and priorities, as well as reinforcing the collaborative nature and interaction that we want to foster.
5. Manage to the Priorities: Everyone has had the situation where a problem or finding crops up, and suddenly there is belief it needs to be the foremost problem we solve. It is a “hair-on-fire” moment, and the belief is that all other work must stop so this can be fixed. While Lean promotes pulling the Andon cord, I like to point out that there are likely many issues in quality when security is involved. I stop everyone in the moment of “hair-on-fire” and ask them to calm down for a minute. Breathe and then look at the list of prioritized items we agreed to work on. I ask if this “hair-on-fire” issue should displace any of those issues. If the answer is yes, we codify it with a risk profile that matches what we did during the Risk Assessment. If it doesn’t (which is almost always the case), we add it to our master list of “all-the-things-we-should-do” so it’s not forgotten.
Result: You recognize the need to fix issues of quality, but also to balance that against where the greatest returns are achieved, and how they align with the company’s objectives. People still feel their concerns are valued, but you also ensure they maintain a balanced and normalized view of priorities.
6. Manage to the Priorities II: Often I have team members who struggle to find the right things to work on. They are concerned about things that are perceived as urgent, and often work on tasks or projects that are urgent, but not necessarily important. I take them through a Steven Covey Four Quadrants exercise. I ask the person what their goal and objective is – what is the mission of their job? Then I ask them to write down in each of the four quadrants things that are Important, Urgent, Not Important, Not Urgent (refer to Steven Covey’s book for how this works). I review with them those tasks that they’ve put in each quadrant and teach them the question, “What happens if this doesn’t get done? Does your mission fail?”. We revise it as they understand what “Important” really means. I then have them tear the piece of paper in half – the top half of Important & Urgent, Important & Not Urgent being pinned to their wall, desk or wherever else they can see it regularly. The bottom half with Not Important & Urgent, and Not-Important & Not-Urgent gets thrown in the trash. The shock and relief in their faces is both amusing and satisfying to see.
Result: The team member learns what is important, that you support them in focusing on important things, and that you are willing to let the “garbage” that doesn’t contribute to goals and quality go in the trash. They feel that their work is now important, they are valued, and they can get rid of the unimportant.
7. Manage the Flow of Work: This effort was far more ad-hoc in many respects, but I drew on numerous methods and tools for managing work.
* Make Work Visible: Kanban – nearly every security team I’ve worked with has preferred Kanban as they way to visualize and manage their work. One task in, one task out, pick up the next task. Because so much of security’s work is intertwined with other teams, it is hard to march to sprint cycles. We could instead weave in and out of activities – pushing things into “on hold”, and flow with any other style of work more easily. What we gained was visibility into what was being worked on, and what was yet to be done.
* Break Projects into Small Achievable Tasks – The problems with “big-bang” large-scale projects are well-known. In addition to the likely failure traits, it also makes team members fearful of such a large effort, or hyper-focused and unable to shift. I mandate that all projects in my team be broken into tasks that they can achieve in two weeks or less and rolled back in two days or less. I ask project leads to break down new projects planning into incremental learning steps. When a solution was arrived at, we broke up rolling out the tools, and configuring them to do the job into separate steps (and even subdivided those two tasks). Each task was designed to be separate and achievable. Most importantly, we could step back (or roll back) from a single task in short order.
* Fit to capacity and level the workload – we monitored the Kanban, as well as I had conversations about people’s workload. If I felt they were being overwhelmed, or if they put in more than 40-45 hours of real work (e.g. I found them in the office after hours all the time), then I would postpone work based on company priorities. I recognized that quality was going to be the first thing that was sacrificed if I didn’t put things on hold (see 'Build for Quality' below). The team now respected that I valued their sanity, would avoid overwork, and would balance priorities. They knew they could do the same. I likewise drove those who didn’t put in the time to deliver. Taking advantage of my desire for quality was not rewarded and would be privately confronted. To quote Nick Galbreath from his time at Etsy: “If you don’t take responsibility, then you probably don’t belong.”
8. Build for Quality: In many projects I have seen people start with a goal, and a deadline of a date. I get frustrated by this model because most people are very bad at estimating how long something will take to accomplish. Even with the concept of an MVP (Minimal Viable Product) they still underestimate the amount of work and time it will require. To overcome this bias, I lay down a set of rules for every project:
* Estimate your timeline and amount of work using the worst-case scenario. We are so over optimistic at time estimation that this will be far more accurate.
* You are allowed to remove features, but you are not allowed to remove quality. If the solution will fail to operate shortly after launch, or there is a probability of disruption to regular operations, go back and fix. Features can be added later. Quality failures are highly disruptive.
* Test, Test, Test. Do the thing and make the change as many times as you want. In test (non-prod) environments. Get good at it. Make mistakes, practice, learn. Then when you get to production, it’s close to rote. You’ve tested all the ways you can think of to fail, and have learned from them for the long term, not just this change.
*If a deadline is going to be missed, evaluate the cost of doing so. Then evaluate the cost of pushing it out with the missing quality (e.g. if it fails every other day, if operations stop, if it gives the wrong answers). Measure this using money and time (which can be equated with money). Things like loss of customers, financial errors, product failures all have monetary costs that can be measured. Deadline pushers will give pause
Result: You will surface over-optimism (it will take time, but you’ll be right more often than the optimists). You will keep a focus on quality, and make sure it stays in the forefront. You’ll encourage learning during testing so that failures are a reward and help avoiding them when they are painful. You’ll also provide a model for everyone to evaluate the impact of quality versus feature deadlines (there is no correct answer until the measure is made).
Applying these techniques can be a challenge to organizations new to these concepts. Facilitation can be the difference between muddling through and finding success. Reach out to Virtual Clarity to see how we can act as that facilitator and send you on your journey….