A NON-DENOMINATIONAL, NON-TECHNICAL TALK

ABOUT PARTS OF DEVOPS CULTURE

BY JON HILDEBRAND (IN NO WAY TO BE CONFUSED WITH GENE KIM)

WHO I AM• Day Job: Senior Cloud Engineer for a US Midwest

based managed service provider (fancy term for I automate data center things)

• Location: Kansas City, Missouri, USA• Free Time: What’s that? Seriously…

• Actually, free time does happen once in a while. I can generally be found doing the following:• Trying to desperately find a baseball game to watch

(GO CUBS GO!) Slightly happy camper about 2016!

• Raising a toddler, which strangely enough, is kind of like managing a technical team sans diapers

• Participating and speaking in technical communities all around (local or not-so-local)

• Just a student of IT practice• Fascinated with the DevOps movement, even though

not in a DevOps environment

THE GOLDEN TRIANGLE

• The constant struggle for IT organizations world wide is effectively managing this triangle

• Pay too much attention to one of the areas and the other two will suffer

• Too often, many problematic to failing IT organizations are too focused the technology angle

WHY ARE WE HERE?

• DevOps, DevOps, DevOps!

• Designed to help balance out the Golden Triangle

• If any of you are like me, you are seemingly fascinated by this topic, even if you aren’t in what you would consider a “traditional application development” environment.

CAMS

• The four main pillars of DevOps

• First coined back in 2010 by John Willis & Damon Edwards

THE PHOENIX PROJECT – THE THIRD WAY

• This book (again)…

• The Third Way – Create a culture of continuous learning and experimentation

• The ultimate goal is to create a high trust culture that reinforces that we are all lifelong learners and that we must take risks

TONIGHT’S TOPIC – CULTURE IN DEVOPS!

• Culture in DevOps gets the short end of the stick

• It’s immensely important to an overall organization

• Without changes to behaviors, a culture may never be properly established, dooming the DevOps movement to failure before even getting to the technology

“You can’t directly change culture. But you can change behavior, and behavior becomes culture.”- Lloyd Taylor, VP Infrastructure, Ngmoco

WHAT WENT SO WRONG WITH IT AND CULTURE?

• Ever hear of Frederick Nietzsche?

• “The Birth of Tragedy from the Spirit of Music”

• The dichotomy Apollonian and Dionysian thought

• Apollonian thought being rational and reasoned

• Dionysian thought being more irrational and chaotic

• Over time, in IT, we’ve applied multiple amounts of rationality and reason to our processes

• Unfortunately, we forget about the chaotic nature of the system we try to control

“Our work is almost always performed within a complex system and how management chooses to react to failures and accidents leads to a culture of fear, which then makes it unlikely that problems and failure signals are ever reported. The result is that problems remain hidden until a catastrophe occurs.” – excerpt from “The DevOps Handbook”

ORGANIZATIONAL CONTROLSOrganizations tend to put methods of controls over various aspects of what we call our day-to-day work. Most organizations will fall under three categories:

• Pathological – Characterized by a hefty amount of fear and mistrust. Information is widely hoarded and withheld for a variety of political reasons. Failure is often hidden from outside view.

• Bureaucratic – Characterized by an extremely high amount of rules and processes designed to protect departmental “turf”. Failure is processed through a system of judgment that results in one of two outcomes: justice or mercy

• Generative – Characterized by actively seeking and sharing information to better enable the organization to achieve its mission. Responsibilities are shared throughout and failure results in reflection and geniune inquiry

YOUR CURRENT SITUATION

Pathological Organizations

Bureaucratic Organizations Generative Organizations

Information Handling

Information is hidden Information may be ignored Information is actively sought

Messenger Handling “Shoot the messenger” Messengers are tolerated Messengers are trained

Responsibilities Shirking of responsibilities Compartmentalized responsibilities

Responsibilities are shared

Team Bridging Highly discouraged Allowed, but generally discouraged

Rewarded

Failure Handling Failure is covered up Treated with either mercy or punishment

Causes inquiry

New Idea Handling New ideas are crushed New ideas create problems New ideas welcomed

Reflect on your own career…

I’ll bet that many have experienced the first two types of organizations…

THE BLAME CULTURE EPIDEMIC• One of the worst tools an

organization can use when handling failure is resorting to “Name, Blame, and Shame”

• Yes, it’s human nature to immediately go that tendency, however, it’s also one of the least productive things that you can do in a crisis

• Emotions are a good thing. Unfortunately, chronic use of this as a practice usually leads to Fear Scary guy, likely saying scary things about you and your loved ones…

TO ERR IS HUMAN

• We all make mistakes... (you don’t say!)

• However, most of the time, our mistakes in our day-to-day work can likely be traced back not to the last action that lead to failure, but to the system that lead to the action

• Blaming the last action would be akin to blaming the axe blade that fell the tree and not the axe itself“Human error is not our cause for troubles; instead, human error is a consequence

of the design of the tools that we gave them.” - Dr. Sidney Dekker

WHY FAILURE IS A GOOD THING“By removing blame, you remove fear; by removing fear, you enable honesty; and honesty enables prevention.” – Bethany Macri, Engineer @ Etsy

• Most DevOps cultures thrive on experimentation, which inherently means failure is going to occur

• Rather than treating failure as the ultimate undoing of a process, it’s used to make a better process

• This leads to the concept of organization learning

• Organization learning is a process of creating, retaining, and transferring within an organization

• Doing so will lead to many personal benefits that greatly benefit the organization• You become more self-diagnosing and self-improving

• You become more skilled at detecting and solving problems

“When response to incidents and accidents are seen as unjust, it can impede safety investigations, promoting fear rather than mindfulness in people who do safety-critical work, making organizations more bureaucratic rather than more careful, and cultivating professional secrecy, evasion, and self-protection.” – Dr. Sidney Dekker

DEVOPS PRACTICES TO CREATE A LEARNING-BASED CULTURE

• Blameless post-mortems• Honest question, how many of you are actually doing post-mortems after events?

• The idea here is to get to the inquiry about the failure, rather than cast the ”name, blame, shame” net upon who did the work in question

• Also great opportunity to gather useful information for the entire organization and published for all to see

• Controlled introduction of failures• Crazy thought…

• This creates many a practice opportunity for inevitable problems

• One core DevOps principle is mastery, which is done through copious amounts of practice

“Coping, fire fighting, and making do were gradually replaced through the organization by a dynamic of identifying opportunities for process and product improvement. As those opportunities were identified and the problems were investigated, the pockets of ignorance that they reflected were converted into nuggets of knowledge.” – Dr. Steven Spear

DEATH TO THE HERO

• Everyone loves the hero, right?

• You can easily identify the ”hero” of the organization by finding one of the most burned out persons

• They generally work extremely long hours, are usually single-handedly troubleshooting issues, and fire fighting to keep services up and running.

• Generally doing this solo

• No shared burden (DevOps core principle is about shared responsibilities)

• Promotes a ton of personal gain at the expense of team effectiveness

“A culture that rewards firefighting breeds arsonists.”

THE TEAM• It can’t be stressed enough that DevOps

culture also promotes many intra-team and inter-team interactions

• Collaboration and affinity are big keys to highly successful DevOps cultures

• Affinity – Process of building inter-team relationships, navigating differing goals and metrics while keeping in mind shared organizational goals and fostering empathy and learning between different groups of people

• Do not forget, that you still have to be able to work with your own team first before working with others

“That’s a team, gentlemen, and either, we heal, now, as a team, or we will die as individuals.” – Tony D’Amato, Any Given Sunday

KEY TAKEAWAYS

• Before you start with DevOps and technology, realize that an entire organization needs to change it’s culture, all the way down to how team members interact

• Without a culture change, most DevOps initiatives are doomed to immediate failure

• While culture change needs to be an organizational imperative, the individuals are ultimately responsible for the behavioral changes need for culture success

• Also, just because DevOps instills this sort of culture doesn’t mean non-development IT organizations can’t either. Many of these points are rather universal across most practices

• Lastly, I may have won a bet by weaving Nietzsche into this presentation!

RECOMMENDED PUBLICATIONS• Highly recommend “The Goal” as a lead in

into “The Phoenix Project”. You’ll recognize many points from “The Goal” in “The Phoenix Project”

• Also, goes without saying since I’m following Gene Kim, that ”The DevOps Handbook” is another great read about things leading up to the technical side of DevOps

• Lastly, I really enjoyed “Effective DevOps” as it had plenty of focus on the managerial side of DevOps teams. You can find plenty of more information about topics not discussed or deeper discussion on silos, effective communication, and developing team trust

THANK YOU AND HAVE A WONDERFUL HOLIDAY SEASON!