devops with smell

DEVOPS WITH SMELLANTONS KRANGA

~ WHOAMI

▸ Full stack developer ~ 15years

▸ Cloud Architect

▸ DevOps evangelist

▸ Innovation Center of Accenture Cloud Platform

▸ Speaker

▸ Marathon runner

ANTONS KRANGA

WHY DEVOPSDEVELOPERS VERSION

DRIVERS FOR SOFTWARE DEVELOPMENT

▸ development price

BEFORE AFTER

▸ innovations speed

DRIVERS FOR SOFTWARE DEVELOPMENT

▸ development price

BEFORE AFTER

▸ innovations speed

▸ specialization silos

▸ ops comes first

▸ cross functional

▸ software defined data centers

▸ complex communication ▸ complexity theory

DEVOPS IS AN ENVIRONMENT WHERE PEOPLE TALK

me :)

DEFINITION OF DEVOPS

SMELL SYMPTOMADEVOPS ANTIPATTERNS

ANTIPATTERN # 0

UNICORNS VS HORSES

You cannot be a unicorn when all others are horses

ANTIPATTERN # 0.1

DEVOPS IN A BOX

You cannot buy culture!

▸ Buy a Golden DevOps LVL5

▸ Become certified DevOps master

▸ Give me DevOps compliance checklist

▸ Give me 5 key DevOps control metrics

ANTIPATTERN # 0.2

REBRANDING

DevOps != Configuration ManagementDevOps != Release Management|DevOps != Product Management…DevOps != (*) Management

DEVOPS IS THE CULTURE YOU CANNOT FIND IN IN ITIL CATALOG

ANTIPATTERN # 1

DEVOPS AS A BLACK BOX

What is the value in DevOps?

! Developer

! Sys-Op

ANTIPATTERN # 2

SORRY, NOT MY DEPARTMENT

Bread ownership and specialization with autonomous teams

vs

http://martinfowler.com/bliki/DevOpsCulture.html

ANTIPATTERN # 2.1

DEVOPS AS A SILO

DevOps teams build their own Silo

▸ You cannot talk to DevOps

▸ Use Jira instead !!!!

▸ RTFM Architecture

ANTIPATTERN # 3

DEFINITION OF DONE

▸ DoD fails with “ilities”

▸ Services can be easily “undone”

ANTIPATTERN # 4

FEAR OF RELEASE

RELEASE === RISK

NO RELEAE NO REVENUE

vs

ANTIPATTERN # 5

SNOWFLAKE SERVER

Applying changes to Server Instance manually leads to unique and distinct server configuration footprint (TECHNICAL DEBT)

http://martinfowler.com/bliki/SnowflakeServer.html

VITAMINSDEVOPS ANTIPATTERNS

VITAMINS

INFRASTRUCTURE AS CODE

ENV

ENV

ENV

DEV …CODE

VITAMINS


DEV

Code is aContract OPS

VITAMINS


DEV

Code is aContract OPS

Put infrastructure into SCM

VITAMINS


DEV

Code is aContract OPS`

Make infrastructure part of app baseline

VITAMINS

WE LIKE CI/CD

▸ We need feedback not to be afraid

▸ Feedback != SPAM

▸ It’s about size of release not frequency

▸ Don’t judge for broken builds

▸ Go home when build is green

PAINKILLERSDEVOPS ANTIPATTERNS

ANTIPATTERN # 6

“JENKINS” DRIVEN DEVELOPMENT

`

`

`

Pipelines are easily becomes your single point of failure

CI

ANTIPATTERN # 6: PROPOSED SOLUTION

SHIFTING RESPONSIBILITY

CI users (DEVs or OPs) are best for managing their jobs

▸ Use DSL to build CI/CD pipelines

▸ Store CI/CD pipelines in git if possible make it part of app baseline

▸ Generate and bootsrap pipelines with API

ANTIPATTERN # 7

SLOW PIPELINES

` ` `

Over-engineered Pipelines

Pipeline execution takes too long

Leads to delayed feedback


SLOW PIPELINES

`

`

`

Parallelize where you can!

ANTIPATTERN # 7

MANUAL PROMOTION

Engineer PRODUATIntegr TestsDEV System Tests

ENVENVENV ENV


KILL SWITCH FOR MANUAL TESTING

PRODUATIntegr TestsDEV System Tests

ENVENVENV ENV

TEST


AB TESTING

PROD A

Integr TestsDEV System Tests

ENV

ENVENV

TEST A

PROD B

ENV

TEST Bfeedback

feedback

measure

ANTIPATTERN # 7.1

CODEREVIEW

PRODUATIntegr Tests System Tests

ENVENVENV ENV

DEV B

feature branch DEV Ccode

review

ANTIPATTERN # 7.1: PROPOSED SOLUTION

REACTIVE CODEREVIEW

ReleaseIntegr Tests System Tests

ENVENV ENV

Full Regression

Calc technicaldebt

ENV

Nightly

DEV A

Standup

code review

…

ANTIPATTERN # 7.1

CODEREVIEW

PRODUATIntegr TestsDEV A System Tests

ENVENVENV ENVmaster

DEV B

feature branch DEV Ccode

review

ANTIPATTERN # 8

DEV DEPLOYMENT VIA CI

DEVENV

DEVENV

DEVENV

…DEV CI



DEV …NO CI

DEVENV

DEVENV

DEVENV



ENV

ENV

DEVENV

…DEV NO CI

WANT TO HACK? SURE!



ENV

ENV

DEVENV

…

YOU BROKE IT? YOU FIX IT!

DEV NO CI

ANTIBIOTICSDEVOPS ANTIPATTERNS

ANTIPATTERN # 9

GOLDEN IMAGE

VM

OS

Problems

▸ Maintained manually

▸ No collaboration

▸ Hard to distribute

▸ Non versioning

Chnorr Service


PROVISIONING

OS Chnorr Service


PROVISIONING

OS

Configure

Infrastructurecode

Harden Download Install

Chnorr Service


PROVISIONING

OS

Configure

Infrastructurecode


cmdb

Chnorr Service

ANTIPATTERN # A

FRAGILE PROVISIONING

OS

Configure

Infrastructurecode


▸ Provisioning success < 100%

▸ Time to provision

Chnorr Service

ANTIPATTERN # A.1

PRIVATE DATA CENTER PROVISIONING

PRIVATE DATA CENTER: NO INTERNET!!!

OS

ConfigureHarden Download Install

Chnorr Service

ANTIPATTERN # A.1



OS


…

apt yum gem

Chnorr Service

ANTIPATTERN # A.1



OS


…

apt yum gem

complicated?

Chnorr Service

ANTIPATTERN # A.1: PROPOSED SOLUTION

STEM CELLS

static dynamic

OS


Chnorr Service


STEM CELLS

OS


LAB PRIVATE DATA CENTER

Chnorr Service


STEM CELLS

OS


LAB PRIVATE DATA CENTER

code

packerPROVISIONSNAPSHOT

docker

Chnorr Service

ANTIPATTERN # B

DEPENDENCY HELL

OS

Configure

Infrastructurecode


▸ Version of libraries

▸ Version of packages

▸ Maintain dependencies

Chnorr Service

ANTIPATTERN # B

DEPENDENCY HELL

OS

Configure

Infrastructurecode


▸ Version of libraries

▸ Version of packages


▸ Version of your infra code


Chnorr Service

ANTIPATTERN # B: PROPOSED SOLUTION

CANARY BUILDS

Accept TestsCI

Unit Test Provision

…

PRECISE version libs

Accept TestsCI Unit Test Provision

…

LATEST version libs

Chnorr Service

Chnorr Service

ANTIPATTERN # C

INFRASTRUCTURE PETAttributes of Pet

▸ Have meaningful names

▸ Long living instance

▸ Often needs manual nursing

▸ Requires scary patching

▸ Leads to snowflakes

▸ PaaS is modern pet

ANTIPATTERN # C: PROPOSED SOLUTION

INFRASTRUCTURE CATTLEAttributes of Pet▸ Have numbers in its name

▸ Short living instance

▸ Immutable configuration

▸ Recreate instead of patching

▸ Requires careful planning


PET TO TRANSFORMATION CATTLE EXAMPLE

Chnorr Service

Database



Chnorr Service

Database

Cattle

Pet



Chnorr Service

Database

Cattle

Pet

User Data

ANTIPATTERN # D

SECRETS LEAK

OS

Configure

Infrastructurecode

Install

cmdbwrong place

for your secretswrong place

for your secrets

Chnorr Service

ANTIPATTERN # D: PROPOSED SOLUTION

SECRETS LEAK

▸ Don’t store secrets with code

▸ Don’t store secrets with configuration

▸ Don’t leave secrets in service

Secrets DON’Ts


SECRETS LEAK

▸ Secret can be leased and rotated

▸ tmpfs is your fiend

Secrets DOs


SECURITY LEASING EXAMPLE

Chnorr Service vault

consul

IAM

Database

api

x hours leasing

AWS

TAKEAWAYSGOOD INFRA CODE

TAKEAWAYS

LEARN PATTERNS BEFORE TOOLS

Patterns Tools

vs

TAKEAWAYS

EVERYTHING MUST HAVE AN API

REST

DSL

CLI

TAKEAWAYS

SELF TESTABLE CODE

▸ Use assertions for infrastructure code

▸ Use acceptance test frameworks

Tests improves your confidence

TAKEAWAYS

READING

▸ Book: A Human Error Approach to Aviation Accident Analysis

▸ Author: Douglas A. Wiegmann Scott A. Shappell

▸ ISBN: 978-0754618737

TAKEAWAYS

READING

▸ Book: Clean Code

▸ Author: Robert C Martin

▸ ISBN: 978-0132350884

THANK YOU

devops with smell

Software