velocity 2011: production begins in development

Production Begins in Development

Lee Thompson Damon Edwards

• CTO Travel/Transportation Products, Hewlett Packard

• Former Chief Technologist, E*TRADE Financial

• Co-writer of “Fully Automated Provisioning” whitepaperLee Thompson

@stagr_lee

• Co-founder DTO Solutions

• DevOps Days organizer

• Co-writer of “Fully Automated Provisioning” whitepaper

Damon Edwards@damonedwards

• Co-founder DTO Solutions

• DevOps Days organizer

• Co-writer of “Fully Automated Provisioning” whitepaper

Damon Edwards@damonedwards

• Consumer-facing web applications

• Webtone utilities

Showing us what is possible...

Clouds

Lean Startup

Continuous Delivery/Deployment

What we want more of...

DevOps!

The Rise of a New IT Operations Support Model

!"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<#="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<#

=" $%? 23 @12=61 $%%% 24;670A6:027.

Why DevOps will emerge:

!)*+,-. 0. 72: >.>611" <40+*7 3425

Why DevOps will not emerge:

!B>1:>461 8967;*. 64* :9* 964<*.: :2

="#$%?#23#@12=61#$%%%#24;670A6:027.C

!)*+,-.#0.#72:#>.>611"#<40+*7#3425#:9*#:2-#<2/7#67<(#:9>.(#56"#=*#524*#*6.01"#688*-:*<#="#DE#2-*46:027.#:*65.C

!B>1:>461#8967;*.#64*#:9*#964<*.:#:2#05-1*5*7:(#67<#)*+,-.#4*F>04*.#6#.0;7030867:#4*:907G07;#23#DE#2-*46:027.#827+*7:02761#/0.<25C

!DEDH#67<#2:9*4#=*.:#-468:08*.#3465*/24G.#64*#68G72/1*<;*<#:2#96+*#72:#<*10+*4*<#27#:9*04#;261.(#*76=107; DE 24;670A6:027. :2 122G 324

!E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9#4*.-*8:#:2#DEDH#67<#2:9*4#=*.:#-468:08*.#3465*/24G.#:96:#0.#614*6<"#688*-:*< /0:907 :9* 07<>.:4" *76=107;#DE#24;670A6:027.#:2#122G#324#

7*/#52<*1.C

!E9*#;42/07;#07:*4*.:#07#:221.#.>89#6.#B9*3(#I>--*:(#*:8C(#/011#9*1-#

688*-:*<#/0:907#:9*#07<>.:4"C

!,-*7#.2>48*#J,KKL#5676;*5*7:#:221.(#/9089#64*#524*#610;7*<#/0:9#:90.#6--42689(#96+*#72:#.**7# -- -

.:05>16:*#<*567<#324#,KKM=6.*<#5676;*5*7:

--.0;7030867:#*7:*4-40.*#564G*:#.964*#:468:027C

DevOps. DevOps. DevOps. DevOps. DevOps.

The Rise of a New IT Operations Support Model

!"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<#="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<#

=" $%? 23 @12=61 $%%% 24;670A6:027.

Why DevOps will emerge:

!)*+,-. 0. 72: >.>611" <40+*7 3425

Why DevOps will not emerge:

!B>1:>461 8967;*. 64* :9* 964<*.: :2

="#$%?#23#@12=61#$%%%#24;670A6:027.C

!)*+,-.#0.#72:#>.>611"#<40+*7#3425#:9*#:2-#<2/7#67<(#:9>.(#56"#=*#524*#*6.01"#688*-:*<#="#DE#2-*46:027.#:*65.C

!B>1:>461#8967;*.#64*#:9*#964<*.:#:2#05-1*5*7:(#67<#)*+,-.#4*F>04*.#6#.0;7030867:#4*:907G07;#23#DE#2-*46:027.#827+*7:02761#/0.<25C

!DEDH#67<#2:9*4#=*.:#-468:08*.#3465*/24G.#64*#68G72/1*<;*<#:2#96+*#72:#<*10+*4*<#27#:9*04#;261.(#*76=107; DE 24;670A6:027. :2 122G 324

!E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9#4*.-*8:#:2#DEDH#67<#2:9*4#=*.:#-468:08*.#3465*/24G.#:96:#0.#614*6<"#688*-:*< /0:907 :9* 07<>.:4" *76=107;#DE#24;670A6:027.#:2#122G#324#

7*/#52<*1.C

!E9*#;42/07;#07:*4*.:#07#:221.#.>89#6.#B9*3(#I>--*:(#*:8C(#/011#9*1-#

688*-:*<#/0:907#:9*#07<>.:4"C

!,-*7#.2>48*#J,KKL#5676;*5*7:#:221.(#/9089#64*#524*#610;7*<#/0:9#:90.#6--42689(#96+*#72:#.**7# -- -

.:05>16:*#<*567<#324#,KKM=6.*<#5676;*5*7:

--.0;7030867:#*7:*4-40.*#564G*:#.964*#:468:027C

DevOps. DevOps. DevOps. DevOps. DevOps.The Rise of a New IT Operations Support Model

!"#$%&'(#)*+,-.#/011#*+21+*#3425#6#7089*#.:46:*;"#*5-12"*<#="#164;*#812><#-42+0<*4.#07:2#6#5607.:4*65#.:46:*;"#*5-12"*<#

=" $%? 23 @12=61 $%%% 24;670A6:027.

Why DevOps will emerge:

!)*+,-. 0. 72: >.>611" <40+*7 3425

Why DevOps will not emerge:

!B>1:>461 8967;*. 64* :9* 964<*.: :2

="#$%?#23#@12=61#$%%%#24;670A6:027.C

!)*+,-.#0.#72:#>.>611"#<40+*7#3425#:9*#:2-#<2/7#67<(#:9>.(#56"#=*#524*#*6.01"#688*-:*<#="#DE#2-*46:027.#:*65.C

!B>1:>461#8967;*.#64*#:9*#964<*.:#:2#05-1*5*7:(#67<#)*+,-.#4*F>04*.#6#.0;7030867:#4*:907G07;#23#DE#2-*46:027.#827+*7:02761#/0.<25C

!DEDH#67<#2:9*4#=*.:#-468:08*.#3465*/24G.#64*#68G72/1*<;*<#:2#96+*#72:#<*10+*4*<#27#:9*04#;261.(#*76=107; DE 24;670A6:027. :2 122G 324

!E9*4*#0.#6#164;*#=2<"#23#/24G#/0:9#4*.-*8:#:2#DEDH#67<#2:9*4#=*.:#-468:08*.#3465*/24G.#:96:#0.#614*6<"#688*-:*< /0:907 :9* 07<>.:4" *76=107;#DE#24;670A6:027.#:2#122G#324#

7*/#52<*1.C

!E9*#;42/07;#07:*4*.:#07#:221.#.>89#6.#B9*3(#I>--*:(#*:8C(#/011#9*1-#

688*-:*<#/0:907#:9*#07<>.:4"C

!,-*7#.2>48*#J,KKL#5676;*5*7:#:221.(#/9089#64*#524*#610;7*<#/0:9#:90.#6--42689(#96+*#72:#.**7# -- -

.:05>16:*#<*567<#324#,KKM=6.*<#5676;*5*7:

--.0;7030867:#*7:*4-40.*#564G*:#.964*#:468:027C

Measuring success...

Quality

Alignment

Cycle Time

Know what kind of business you are

What is your tolerance for...

Risk

Change

Consumer-facing applications...

"Move fast and break things. Unless you are breaking stuff,

you are not moving fast enough."-Mark Zuckerberg

Consumer-facing applications...

"Move fast and break things. Unless you are breaking stuff,

you are not moving fast enough."-Mark Zuckerberg

Webtone utilities...

Webtone utilities...

Webtone utilities...

Webtone utilities...

Webtone utilities...

What do you want...

What do you want...

What do you want...

What do you want...

What do you want...

What do you want...

What do you want...

What do you want...

Quality

Alignment

Cycle Time

Clouds

DevOps

Lean Startup

Continuous Delivery/Deployment

It all starts in Development

It all starts in Development

But what do we tell them to do and how do we get them to do it?

Share ownership of availability

• Developers must wear pagers

• Incident command training so everyone knows their roles

• Notification mechanism? Access provisioning?

Non-functional requirements are first class citizens

• All of the “-ility” requirements

• As much a part of your service as the businessʼs functional requirements

• Manage through same process

• Make sure they are on the schedule

• Donʼt over commit

Strive for parity between dev & prod

• Develop test data fixtures for all environments

• Implement Mock Services for major infrastructure pieces for Developer user.

• Continuous integration means integrate early

• Use the deployment, config, and packaging tools in dev

Push config management discipline back to Dev

• Dev is about creating variation, Ops is about eliminating variation

• Augment deployment toolchain to support the variation

• Do developers use the tools?• Do developers like them?• Devʼs need config too• Accept config contributions and patches

from dev

Packaging... itʼs not just for the OS• If you want to be a high performing web

operations organization, you need to take change management seriously.

• Strict versioning is needed, and package managers do that.

• Itʼs about being idempotent.

• Transfer packaging responsibility to dev

• Define the packaging constructs you will support

http://webscalebook.com

Config is code• if itʼs code it needs to be managed like

code...

• Should be transparent and identical SDLC in both dev and ops

• Avoid or eliminate asymetric release processes

Tailor release artifacts to roles• “Small teams make better software”• Make sure one team getting stuck doesnʼt

prevent another team to release (org coupling)

• Large codebases suffer software entropy effects

• Build an infrastructure that can reliably manage lots of smaller artifacts

• Org conflict is a good time to suggest breaking up a codebase into separate concerns

Standard management vocabulary

• Consistent and expected management behavior

• Across components and releases

• “start, stop, status, update, install...”

Rollback• Rollback that works

• No, really works

• Seriously, really really works

• You say it will work? Test it right now

Standard metrics abstractions• Dev surface metrics to Ops

• Avoid the back and forth conversations

• Use a standard framework

• Use standard types (gague, counter, timer...)

• Ops knows what to expect and how to visualize

https://github.com/codahale/metrics

Push test ownership to the edges• QA is “Quality Assurance”

• QA writing tests = bottleneck

• QA writing tests = avoiding responsibility

• QA is architecture / tooling / assurance

• Test Driven Development

• Test Driven Operations (yes, you too)

• Bottom line: Everyone owns quality.

Test outside of the box

Crowd Test Simulate

Simulation

AUT

Build the right lifecycle...

Quality

Alignment

Cycle Time

Continuous Delivery...

Continuous Delivery...

AutomatedTesting

AutomatedRelease

Deployment...

Donʼt be so dogmatic

Decoupled Release Model Example

Repo

Audit

Release

ReleaseRepoA

OS

services

app code

ReleaseRepoC

ReleaseRepoB

Sandbox

Validate

Release

"Check for conflicts and

run validation scripts"

Package Manifest

SouceRepo

"Check-in"

Promote

Job

ReleaseRepoA

ReleaseRepoC

ReleaseRepoBProd

Repo

"Promote release"

1

2

3

4

Hudson

ReleaseRepoB

ReleaseRepoC

ReleaseRepoA

"Determine what packages

and versions are available"

ProdReleaseRepo

Prod

Chef

RightScale

Rundeck

"yum install"

Deploy

Job

5

Bootstrap Develop Release Operate

Prod

Chef

RightScale

Rundeck

Monitor

Jobs &

Adhoc

CommandsOperate

1

2

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Rundeck

Instance

CI Server

Instance

Staging

Release

Repo

Prod

Release

RepoDev Test

Monitoring

& Testing

Operations

Rundeck

"Auto-Deploy

New App

Infrastructure

1Source

Repo

Chef

Instance

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3Automatic

Deployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Repo

Audit

Release

ReleaseRepoA

OS

services

app code

ReleaseRepoC

ReleaseRepoB

Sandbox

Validate

Release

"Check for conflicts and

run validation scripts"

Package Manifest

SouceRepo

"Check-in"

Promote

Job

ReleaseRepoA

ReleaseRepoC

ReleaseRepoBProd

Repo

"Promote release"

1

2

3

4

Hudson

ReleaseRepoB

ReleaseRepoC

ReleaseRepoA

"Determine what packages

and versions are available"

ProdReleaseRepo

Prod

Chef

RightScale

Rundeck

"yum install"

Deploy

Job

5

Bootstrap Develop Release Operate

Prod

Chef

RightScale

Rundeck

Monitor

Jobs &

Adhoc

CommandsOperate

1

2

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Rundeck

Instance

CI Server

Instance

Staging

Release

Repo

Prod

Release

RepoDev Test

Monitoring

& Testing

Operations

Rundeck

"Auto-Deploy

New App

Infrastructure

1Source

Repo

Chef

Instance

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3Automatic

Deployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Decoupled Release Model Example

Bootstrap

Rundeck

Instance

CI Server

Instance

Staging

Release

Repo

Prod

Release

RepoDev Test

Monitoring

& Testing

Operations

Rundeck

"Auto-Deploy

New App

Infrastructure

1Source

Repo

Chef

Instance

Repo

Audit

Release

ReleaseRepoA

OS

services

app code

ReleaseRepoC

ReleaseRepoB

Sandbox

Validate

Release

"Check for conflicts and

run validation scripts"

Package Manifest

SouceRepo

"Check-in"

Promote

Job

ReleaseRepoA

ReleaseRepoC

ReleaseRepoBProd

Repo

"Promote release"

1

2

3

4

Hudson

ReleaseRepoB

ReleaseRepoC

ReleaseRepoA

"Determine what packages

and versions are available"

ProdReleaseRepo

Prod

Chef

RightScale

Rundeck

"yum install"

Deploy

Job

5

Bootstrap Develop Release Operate

Prod

Chef

RightScale

Rundeck

Monitor

Jobs &

Adhoc

CommandsOperate

1

2

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Rundeck

Instance

CI Server

Instance

Staging

Release

Repo

Prod

Release

RepoDev Test

Monitoring

& Testing

Operations

Rundeck

"Auto-Deploy

New App

Infrastructure

1Source

Repo

Chef

Instance

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3Automatic

Deployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Decoupled Release Model Example

Develop

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Repo

Audit

Release

ReleaseRepoA

OS

services

app code

ReleaseRepoC

ReleaseRepoB

Sandbox

Validate

Release

"Check for conflicts and

run validation scripts"

Package Manifest

SouceRepo

"Check-in"

Promote

Job

ReleaseRepoA

ReleaseRepoC

ReleaseRepoBProd

Repo

"Promote release"

1

2

3

4

Hudson

ReleaseRepoB

ReleaseRepoC

ReleaseRepoA

"Determine what packages

and versions are available"

ProdReleaseRepo

Prod

Chef

RightScale

Rundeck

"yum install"

Deploy

Job

5

Bootstrap Develop Release Operate

Prod

Chef

RightScale

Rundeck

Monitor

Jobs &

Adhoc

CommandsOperate

1

2

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Rundeck

Instance

CI Server

Instance

Staging

Release

Repo

Prod

Release

RepoDev Test

Monitoring

& Testing

Operations

Rundeck

"Auto-Deploy

New App

Infrastructure

1Source

Repo

Chef

Instance

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3Automatic

Deployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Decoupled Release Model Example

Repo

Audit

Release

ReleaseRepoA

OS

services

app code

ReleaseRepoC

ReleaseRepoB

Sandbox

Validate

Release

"Check for conflicts and

run validation scripts"

Package Manifest

SouceRepo

"Check-in"

Promote

Job

ReleaseRepoA

ReleaseRepoC

ReleaseRepoBProd

Repo

"Promote release"

1

2

3

4

Hudson

ReleaseRepoB

ReleaseRepoC

ReleaseRepoA

"Determine what packages

and versions are available"

ProdReleaseRepo

Prod

Chef

RightScale

Rundeck

"yum install"

Deploy

Job

5

Release

Repo

Audit

Release

ReleaseRepoA

OS

services

app code

ReleaseRepoC

ReleaseRepoB

Sandbox

Validate

Release

"Check for conflicts and

run validation scripts"

Package Manifest

SouceRepo

"Check-in"

Promote

Job

ReleaseRepoA

ReleaseRepoC

ReleaseRepoBProd

Repo

"Promote release"

1

2

3

4

Hudson

ReleaseRepoB

ReleaseRepoC

ReleaseRepoA

"Determine what packages

and versions are available"

ProdReleaseRepo

Prod

Chef

RightScale

Rundeck

"yum install"

Deploy

Job

5

Bootstrap Develop Release Operate

Prod

Chef

RightScale

Rundeck

Monitor

Jobs &

Adhoc

CommandsOperate

1

2

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Rundeck

Instance

CI Server

Instance

Staging

Release

Repo

Prod

Release

RepoDev Test

Monitoring

& Testing

Operations

Rundeck

"Auto-Deploy

New App

Infrastructure

1Source

Repo

Chef

Instance

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3Automatic

Deployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Decoupled Release Model Example

Prod

Chef

RightScale

Rundeck

Monitor

Jobs &

Adhoc

CommandsOperate

1

2

Operate

Repo

Audit

Release

ReleaseRepoA

OS

services

app code

ReleaseRepoC

ReleaseRepoB

Sandbox

Validate

Release

"Check for conflicts and

run validation scripts"

Package Manifest

SouceRepo

"Check-in"

Promote

Job

ReleaseRepoA

ReleaseRepoC

ReleaseRepoBProd

Repo

"Promote release"

1

2

3

4

Hudson

ReleaseRepoB

ReleaseRepoC

ReleaseRepoA

"Determine what packages

and versions are available"

ProdReleaseRepo

Prod

Chef

RightScale

Rundeck

"yum install"

Deploy

Job

5

Bootstrap Develop Release Operate

Prod

Chef

RightScale

Rundeck

Monitor

Jobs &

Adhoc

CommandsOperate

1

2

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Rundeck

Instance

CI Server

Instance

Staging

Release

Repo

Prod

Release

RepoDev Test

Monitoring

& Testing

Operations

Rundeck

"Auto-Deploy

New App

Infrastructure

1Source

Repo

Chef

Instance

AutomaticDeployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3Automatic

Deployment

WatchCode

Build

UnitTests

PublishResults

Continuous

Deployment

LoopIntegration Testing

Source

Repo

Package Manifest

Code, Config, Control

"Check-in"

Develop

"Check-in"

Dev Test

"Check-out"

"Verify"

Release

Repo

A"Promote"

1

2

3

Decoupled Release Model Example