build & release engineering

ì Build & Release Engineering

By: Pranesh Vi.al CG h.p://linkedin.com/in/praneshvi.al

Main Agenda of this session

h"p://linkedin.com/in/praneshvi"al

2

Key Takeaways from this session ì  So<ware Release Principles & its importance.

ì  Not to expect results from “Day 1”. It’s a slow process. Takes months or at Lmes even years to reach perfecLon levels.

ì  What is “ConLnuous IntegraLon”?

ì  Steps associated with “ConLnuous IntegraLon”?

ì  What is a “ConLnuous Delivery”?

ì  Steps associated with “ConLnuous Delivery”?

ì  What is a “ConLnuous Deployment”?

ì  Release Engineering Steps.

ì  Main Focus of Release Engineering. h"p://linkedin.com/in/praneshvi"al

3

Key Takeaways from this session ì  ConfiguraLon Management.

ì  Principles of Managing ApplicaLon ConfiguraLon.

ì  Its not just about the tools. Its about the big “Cultural Change” that everyone has to go through .

ì  Release Engineering Architecture.

ì  Deep dive into commit / component builds.

ì  AnL-‐Pa.erns in commit / component builds.

ì  Aim of Deployment Pipeline.

ì  AnL-‐Pa.erns of Deployment Pipeline.

ì  Best qualiLes of a Deployment Pipeline.


4

Key Takeaways from this session ì  Deployment on Large Scale ProducLon Environments.

ì  Why Automated Deployment is an indispensable goal ???

ì  Deep dive into TesLng Quadrant.

ì  Deep Dive into Automated Acceptance Tests.

ì  Few more Generic AnL-‐Pa.erns.

ì  AdapLon of Release Engineering.

ì  Tools Used at each of the stages.

ì  Managing Environments.

ì  Managing Releases of complex systems. h"p://linkedin.com/in/praneshvi"al

5

Key Takeaways from this session

ì  Steps involved to build a deployment pipeline.

ì  Everyone is an equal contributor to reach “ConLnuous Deployment” stage.

ì  MulL-‐Tenancy Cloud.

ì  MulL-‐Instance ApplicaLons.

ì  Zero DownLme Deployment.

ì  ExtracLng the server logs.

ì  Next generaLon tools in the DevOps world.

ì  PreparaLon for the Release.

ì  Maturity of a “Deployment Pipeline”.


6

Target Audience

ì  Product Managers / Business Analysts.

ì  Engineering Managers.

ì  Development Teams.

ì  Quality Engineering Teams.

ì  System Admins.

ì  Service Engineering / ProducLon Engineering Teams.

ì  On-‐site Coordinators.


7

Software Release Principles & its importance.

ì  Should be Fast / Predictable / Repeatable.

ì  Automate almost everything.

ì  Keep everything in Version Control.

ì  If it hurts, do it more o<en.

ì  Improve the quality of the Builds. Push the boundaries of test automaLon so that quality is not compromised.

ì  Done means the so<ware is released and has reached the producLon.


8

What is Deployment?

ì  Run the same commands in several hosts.

ì  Install the required so<ware from a central repository so that the end state of every host is same.

ì  Based on the host-‐type, install the required packages.

ì  End goal is to bring all the hosts of the respecLve host-‐type in the same state.


9

What is “Continuous Integration”?

ì  Building, deploying and tesLng the applicaLon.


10

Steps associated with “Continuous Integration”?

ì  Integrate all the components at regular intervals.

ì  Binaries built only once and used in all the environments.

ì  Execute Unit Tests before the packages are generated.

ì  Deploy the latest packages onto an INT environment.

ì  Run the automated tests, validate the test results.

ì  Track the Code quality criteria such as staLc code analysis, test coverage.

ì  Deploying to other environments with a pre-‐defined frequency.

ì  Repeat the process.

ì  It’s a pracGce, not a tool


11

What is a “Continuous Delivery”?

ì  Building, deploying, tesLng, promoLon of the applicaLon to the next environment.

ì  Image credit: h.p://www.axonivy.com/


12

Steps associated with “Continuous Delivery”? ì  Steps followed in “ConLnuous IntegraLon”.

ì  AutomaLc promoLon / deployment to other environments (QA, Staging, Performance etc…) upon successful execuLon of automated tests.

ì  ApplicaLon is always ready to deploy to producLon through a largely automated process.


13

What is a “Continuous Deployment”?

ì  Building, deploying, tesLng, promoLon of the applicaLon to the producLon.

ì  Image credit: Yassal Sundman


14

Release Engineering Principles & its importance.

ì  Minimize Cycle Time from check-‐in to producLon push.

ì  Everybody is responsible for the Delivery Process.

ì  ConLnuous Improvement.

ì  AcLviLes such as SCM, Release planning, Automated Deployment, Acceptance TesLng, Performance TesLng are NOT SECONDARY.

ì  Generates no revenue Lll its in the hands of the end-‐users.

ì  Customer SaLsfacLon through early & conLnuous delivery of so<ware.


15

Release Engineering Steps

ì  SCM (Git, SVN, Clearcase).

ì  Automated Commit / Component Builds (Generate Binaries).

ì  Generate Code Quality Metrics.

ì  Build Deployment Pipeline.

ì  Automated Acceptance TesLng.

ì  Automated PromoLon to subsequent Environments.

ì  Single click or automated push to producLon.


16

Main Focus of Release Engineering

ì  Build -‐> Deploy -‐> Test -‐> Release

ì  Every single check-‐in should potenLally be in a releasable state.


17

Configuration Management

ì  Using Version Control and commit everything.

ì  Check-‐in the changes at regular frequency.

ì  Use meaningful messages for every commit.

ì  Managing Dependencies (External Libraries, Components).

ì  Managing the Infrastructure (Consistent network topology, firewall, OS configuraLon, patches etc... across all the environments).

ì  Managing the Environments & the tools to be used for the same.

ì  Managing the Change Process.


18

Principles of Managing Application Configuration

ì  Good understanding of the stage in which the configuraLon should be injected (Assembly, Deployment, Restart etc…)

ì  ConfiguraLon sefngs for the applicaLon to be in the project’s root directory.

ì  Should always be automated and values checked into repository.

ì  Use clear naming convenLon.

ì  Avoid Over-‐engineering. Keep it as simple as possible.

ì  Ensure that the configuraLon is tested properly a<er deployment.


19

Release Engineering Architecture

ì  Its all about building the “Build & Release” pipeline.

ì  Built using Jenkins or similar ConLnuous IntegraLon Tools.

ì  IllustraLon of a “Delivery Pipeline” (Image Courtesy: “ConLnuous Delivery” by Jez Humble & David Farley).


20

Deep dive into commit / component builds:

ì  Aim: ì  Eliminate builds that are unfit for producLon.

ì  Steps: ì  Compile the code. ì  Run a set of commit tests. ì  Run Lint based tests or some of the basic staLc code analysis tasks. ì  Publish the results. ì  Once above steps are done, build the package and upload the binary to

a centralized repository. ì  Add tests on an ongoing basis. ì  Keep a watch on the build execuLon Lme and opLmize frequently. ì  Explore the ‘Pre-‐Commit’ or ‘Pre-‐Flight’ build opLons provided by some

of the CI tools.


21

Anti-‐Patterns in commit / component builds:

ì  Don’t check-‐in new code on a broken build. The only fix that has to go are the changes that fix the broken build.

ì  Always run commit tests locally before the commit.

ì  Always wait for commit tests to pass before next check-‐in.

ì  Developers who have commi.ed their code are responsible for the build process to succeed.

ì  Never go home on a broken build.

ì  Time-‐box during every failed build. Give about 10-‐20 minutes to fix, else rollback the changes.

ì  Don’t EVER comment failing tests.

ì  Take ownership for all the breakages that result from your changes and fix them accordingly.


22

Aim of Deployment Pipeline ì  Every Step should be visible (Results in be.er collaboraLon).

ì  Deliver useful, working so<ware to the users as early as possible.

ì  Early feedback (IdenLfy & resolve issues in the early stages).

ì  Enable teams to deploy & release any version of their so<ware at any point to any of the environments.

ì  Avoid last day heroics on the release day.

ì  Release in small chunks (Avoid “Big Bang” release).

ì  Should be driven by tools & not by sviduals.

ì  Ability for every change to be transparent & propagate them through the pipeline.

ì  Ability to roll-‐back to a previous stable state in case of any failures. h"p://linkedin.com/in/praneshvi"al

23

Anti-‐Patterns of Deployment Pipeline ì  Manual Deployment of So<ware.

ì  Deployment performed by mulLple teams.

ì  The order of steps are not defined.

ì  No proper “Run books” for failures.

ì  Too much of reliance of manual tesLng.

ì  CorrecLon to the release process during the actual producLon release.

ì  ConfiguraLon across all the environments varies to a large extent.

ì  Manual ConfiguraLon management of ProducLon Environment.

ì  Deployment to a producLon-‐like environment is done only development is 100% complete.


24

Best qualities of a Deployment Pipeline

ì  Deployments should be automated.

ì  Deployments should be done frequently (If possible, for every single check-‐in).

ì  Provide early feedback so that the development team can act on the failures in a faster way.

ì  Good AutomaLon Coverage to test early.

ì  Should be scalable.

ì  Should enable one-‐click deployment.


25

Deployment on Large Scale Production Environments

ì  What are host-‐types?

ì  What are recipes / cookbooks?

ì  Tools like Pogo OR other agent based deployment tools.

$ ssh <target-‐host-‐name> 'whoami;’

praneshv


26

Why Automated Deployment is an indispensable goal ???

ì  On most occasions, the deployment documentaLon is outdated.

ì  Logging all the steps in the form of scripts is very beneficial for book keeping purposes.

ì  Works seamlessly if all the steps are taken care of.

ì  Even non-‐experts should be able to deploy effortlessly.

ì  Every team using automated deployment scripts results in its maturity & less prone to errors.

ì  Take off the dependency from the deployment expert and uLlize them for more challenging tasks.

ì  Its fast, cheap. Facilitates in early tesLng & faster feedback cycles.

ì  Deployment Logs are auditable for future references.


27

Deep dive into Testing Quadrant

Image Credit: Concept defined by ‘Brian Marick’ (Diagram obtained from “ConLnuous Delivery” by ‘Jez Humble’ & ‘David Farley’.


28

Deep Dive into Automated Acceptance Tests

ì  Aim: ì  Avoid manual tests to whatever extent possible.

ì  Steps to be followed: ì  Tests to be performed in producLon-‐like environment. ì  If sefng up environment is expensive, use a scaled-‐down

environment. ì  Setup the actual user’s environment on a grid. ì  Use Business language in the scripts instead of technical jargon

(‘Place Order’ instead of ‘Clicking on bu.on XYZ’). ì  Well-‐defined exit criteria for Pass / Fail of tests. ì  Don’t fail the test due to minor issues.


29

Deep Dive into Automated Acceptance Tests continued…

ì  Steps to be followed: ì  Include a few tests to assert external systems. ì  If you don’t care about a parLcular field, don’t bother tesLng it. ì  These tests should be run a<er every deployment. ì  Block the pipeline when there are massive failures. ì  Parallelize the tests to whatever extent possible.

ì  Outcome of this step are the so<ware packages that have fought against all the tests & challenges like a mythical hero and ready to take on the world !!!


30

Few more Generic Anti-‐Patterns

ì  TesLng done on developer machines.

ì  Service Engineering team hasn’t even seen the applicaLon Lll the actual release date.

ì  Installers, ConfiguraLons etc… not done Lll the release date.

ì  No collaboraLon between development team & SE teams.

ì  Late setup of Staging Environment.

ì  Release too many changes, unable to figure out what caused the failure.

ì  Changing the configuraLons directly on ProducLon o<en resulLng in disasters.


31

Adaption of Release Engineering ì  Change in the mindset of the team members.

ì  All aspects of development, tesLng, staging, producLon environments (most importantly configuraLon management) to be managed through a VCS.

ì  Integrate development, tesLng, release teams as a part of the development process.

ì  Manage the Infrastructure to build environments in no Lme.

ì  Beef up the test automaLon coverage so that there’s not much reliance on Manual tesLng.

ì  Rehearse the release & rollback process so that its easy to do it Lme & again.

ì  Reach a stage wherein “So<ware Release” is in self-‐service mode.


32

Tools Used at each of the stages. ì  Version control – Git

ì  Build Tools – Ant, Nant, MSBuild, gmake, Maven, Buildr, Psake.

ì  Agile / Scrum – Jira.

ì  StaLc validaLon – Coverity, SonarQube

ì  Unit tesLng – junit

ì  Test automaLon – Protractor, Selenium

ì  ConLnuous IntegraLon – Jenkins, Atlassian Bamboo, Thoughtworks Go, UrbanCode AntHillPro

ì  Deployment – Pogo, Chef

ì  Environment provisioning – Puppet, Chef

ì  IAAS, PAAS – AWS, Azure, Docker, Vagrant h"p://linkedin.com/in/praneshvi"al

33

Tools Used at each of the stages.

ì  Image Credit : h.p://maestrodev.com


34

Managing Environments ì  No ApplicaLon is an Island.

ì  Poor ConfiguraLon Management results in significant waste of Lme, addiLonal costs, tech debt etc…

ì  Should always be cheaper to create a new environment than repairing a broken environment.

ì  Few of the names used for Environments ì  Development ì  IntegraLon ì  QA / TesLng ì  Staging / Pre-‐ProducLon ì  Performance ì  Canary / Bucket TesLng ì  ProducLon


35

Managing Releases of complex systems


36

Image Credit: Slide deck available at h.ps://learn.chef.io/

Managing Releases of complex systems


37

OrchestraLon: ì  Automated arrangement, coordinaLon, and management of complex computer systems,

middleware, and services. ì  Aligning the business request with the applicaLons, data, and infrastructure

ì  Deployment Rhythm

ì  Its about gefng the perfect configuraLon, checking in these values and automate the release using these values.

ì  Have the right sequence of steps in the configuraLon file.

ì  Have the right kind of checks & balances at every stage.

ì  PracLce roll-‐backs in case of issues.

ì  Implement fail-‐safe methodologies in case of issues.

ì  Build environments wherein the ProducLon traffic can be replayed (Use of access logs to simulate user acLons).

Steps involved to build a deployment pipeline

ì  Model your value stream & create a walking skeleton.

ì  Automate the build & deployment process.

ì  Automate unit tests & code analysis.

ì  Automate Acceptance Tests.

ì  Automate Releases.


38

Multi-‐Tenant Applications ì  Customers share the same cloud plavorm and infrastructure and their data

is commingled.

ì  Single code-‐base for the applicaLon.

ì  Upgrades are executed easily by the SaaS provider.

ì  All the tenants are using the same version.

ì  Data of different tenants is stored in the same place, however, measures taken to make it inaccessible to other tenants.

ì  BuckeLze the access based on the IP Range / Cookie range / Header InformaLon etc...

ì  Toggle a Feature using configuraLon.

ì  Hard to maintain different versions for different tenants. h"p://linkedin.com/in/praneshvi"al

39

Multi-‐Instance Applications

ì  MulLple customers run their own separate instance of an applicaLon and OS running on a separate VM.

ì  Avoid comingling of data.

ì  Develop & use their own logical piece of the cloud service.

ì  Allows for greater flexibility and control of configuraLon, customizaLon, updates and upgrades.

ì  Ability to migrate instance to an on-‐premise server, or to another cloud hosLng provider


40

Zero Downtime Deployment

ì  What is ‘Zero DownLme Deployment’?

ì  What is ‘Out Of RotaLon’ (OOR) ?

ì  What is ‘Concurrency’ ?

ì  Colo (Data-‐Center) based deployments.


41

Extracting the server logs ì  Standardize the logging messages. Involves lot of effort from

the development teams.

ì  Setup Splunk alerts for ‘FATAL’ errors in the code.


42

Next generation tools in the DevOps world

ì  Chef

ì  Puppet

ì  Cfengine

ì  Salt


43

Preparation for the Release

ì  Any issues during the release, stop or postpone the release. Stability takes the highest priority.

ì  Prepare Release Plan or Runbook with correct sets and if possible automate them.

ì  IdenLfy the error-‐prone steps and automate them.

ì  Perform the drill for performing rolling back of the releases.

ì  Should be as simple as selecLng a value and clicking a bu.on.

ì  Let one team perform all the releases. (Too many cooks spoil the broth).


44

Maturity of a “Deployment Pipeline” ?


45

ì  Image Credit: h.p://www.praqma.com/

Q & A


46

Thank You


47

References:

ì  Details about each of the phases from “ConLnuous Delivery” by Jez Humble, David Farley

ì  Source of some of the generic images used – “Unknown”.

ì  RespecLve credits given to the images used in the same slide.

ì  h.ps://gigaom.com/2014/01/26/mulL-‐tenant-‐or-‐mulL-‐instance-‐cloud-‐lets-‐do-‐both/

ì  h.p://diginomica.com/2013/12/20/mulL-‐tenant-‐mulL-‐instance-‐saas-‐spectrum/

ì  h.ps://msdn.microso<.com/en-‐us/library/hh534478.aspx

ì  h.ps://www.youtube.com/watch?v=CE4hLYhfmBE -‐ Deployment using pogo.


48