Application Automation with HabitatSydney Cloud Native Meetup

August 22, 2017

Matt RayManager, Solutions Architect – APJChef Softwarematt@chef.io@mattray

Consumption Models have Changed

● Standard “IT Shared Services” approach no longer viable

● Dev teams have spoken; they want agility and self-service

● Various dev teams assemble their own toolchains to move quickly

● Thrashing on toolchains prevents cross-organizational success

● Traditional centralized decision making isn’t going to work

Problems with Shared Services

● Slow

● Inflexible

● Enforce “control” vs enabling “access”

● Creates a barrier that slows down time to market

Consumption Models

Old Create a helpdesk ticket Send an email Submit a Change Request

New API-driven interaction SDK for developers to build on top of APIs CLI tools for ease of use Slack (searchable real-time communication)

LOBLOB

FOCUS ON SPEED

Tension caused by the demands placed on teams…

…can be resolved by vertical integration and automation…

…to deliver a future of developer services and software at speed

SHARED SERVICES

Vertical Integration is key to velocityA balance of increasing speed, improved efficiency and decreasing risk

FOCUS ON RISK

SHARED SERVICES

LOBLine of

Business

LOBLine of

BusinessLOB

SHARED SERVICES

LOB

DEVELOPER SERVICESBUILD • DEPLOY • MANAGE • COLLABORATE

”Business Value with Developer ServicesShifting capabilities to match business requirements

Developer Services EngineerLine of Business Development TeamI provide services that developers and development teams use to build and deliver applications.

Developer Services TeamsTraditional Central IT

System AdministratorCentralized Enterprise IT TeamI manage and deliver infrastructure required to run software in my organization.

Enabling development teams to get insights into speed, efficiency and risk of delivery of their software

Reducing risk to my organization from my infrastructure and software that runs on it

Enabling development teams to ship software at speed while maintaining quality and reducing risk

Reliably managing changes to infrastructure requirements

Providing on-demand, self-service infrastructure and services tailored to developer needs

Managing and lowering costs of running, configuring, and maintaining infrastructure

DEPLOY

MANAGE

Don’t measure me on traditional IT metrics, but on the metrics of the businessJim Fowler, CIO, GE Capital

BUILD

Reducing friction by enabling full stack transparency and management capability with separation of duties

Delivering configured infrastructure and handing off responsibility to development teams COLLABORATE

Modern Architecture Trends

Modern Architecture Principles

12 Factor Apps/Cloud NativeMicroservicesServerlessContainers

12 Factor/Cloud Native

https://12factor.net/

“The twelve-factor app is a methodology for building software-as-a-service apps that:● Use declarative formats for setup automation, to minimize time and cost for new developers joining the project;● Have a clean contract with the underlying operating system, offering maximum portability between execution environments;● Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration;● Minimize divergence between development and production, enabling continuous deployment for maximum agility;● And can scale up without significant changes to tooling, architecture, or development practices.”

Microservices

“…the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms…”

- Martin Fowler and James Harris

https://martinfowler.com/articles/microservices.html

Microservices Characteristics

● Componentization via Services● Organized around Business Capabilities● Products not Projects● Smart endpoints and dumb pipes● Decentralized Governance● Decentralized Data Management● Infrastructure Automation● Design for failure● Evolutionary Design

Serverless

“Serverless architecture replaces long-running virtual machines with ephemeral compute power that comes into existence on request and disappears immediately after use.”

- Thoughtworks Tech Radar, March 2017

Containers

Package the application and dependencies in a deployable formatReduce/remove dependency on underlying OS/InfraSmall Area of ConcernSingle process

Common themes

Reduced Area of ConcernService IndependenceInfrastructure IndependenceDecentralized Management

Monolith ServerlessMicroservicesA

pp

licat

ion

Ap

plic

atio

nS

erv

ice

s

Centralized Highly Distributed

Monolith ServerlessMicroservicesA

pp

licat

ion

Ap

plic

atio

nS

erv

ice

s

Few Large Artifacts Thousands of Small Artifacts

Complexity Cliff

Scale Increases - Number of services under management increases

Efficiency Declines - Number of ways to manage each service is different

Risk Exposure Increases -Confidence that each service is secure, compliant, updated, etc.

Developer Services

“The services required for application developers to effectively run applications. This consists of the underlying infrastructure, compute runtimes, language runtimes, and application services to persist and retrieve data. These services should be available via a common API, with a focus on making the service easy to consume.”

- Me

Application Services

What Developers Need

Application Runtimes

Habitat is automation for Application Runtimes & Application Services to deliver Cloud Native Developer Services

Building Developer Services With Habitat

Lifecycle of any software

Build Deploy Run

Build Lifecycle

The dreaded build cycle

wget https://some.place/package.tar.gz

tar xfzv package.tar.gz

cd package

./configure

make

make install

The dreaded build cycle

The dreaded build cycle

Habitat Defines the Build Lifecycle

● Default implementation is C/C++ Build Lifecycle

● Other default implementations provided by Scaffolding

○ core/scaffolding-ruby

○ core/scaffolding-node

○ core/scaffolding-go

● Build dependencies explicitly declared

● Runtime dependencies explicitly declared.

Habitat Artifacts

● Dependencies declared

○ Libraries

○ External Services

● Services/Port-bindings Exported

● Signed

Habitat Build Cycle

USER ARTIFACTPLAN DEPOT

DEMO

Deploy Lifecycle

Deploying Artifacts

SERVICE

SUPERVISOR

SERVICE

SUPERVISOR

SERVICE

SUPERVISOR

SERVICE

SUPERVISOR

DEPOT ARTIFACT

BARE METAL

CONTAINERS

AMI

VM

Libraries

Operating System

Application

Traditionallythe domain of the OS Vendor

Application Libraries

Config Changes

Bot

tom

to T

op D

epen

denc

y R

esol

utio

n

Application &Libraries

Minimized OS

Top

to B

otto

m D

epen

denc

y R

esol

utio

n

Export Deployable Artifacts

● Docker

● ACI

● Mesos

● TAR

Run Lifecycle

What do applications need to run?

What do applications need to run?

• Process Supervisor• Lifecycle events

• Start, Stop, Reconfigure, etc

• Environment specific configuration• Knowledge of peers• Knowledge of dependent services

The Habitat Supervisor

SERVICE

SUPERVISOR

SERVICE

SUPERVISOR

SERVICE

SUPERVISOR

SERVICE

SUPERVISOR

SERVICE

SUPERVISOR

SERVICE

SUPERVISOR

Lifecycle Hooks

• init• run• health_check• file_updated• reload• reconfigure• post-run• suitability

Peers

Service Groups

Gossip

Availability increases with

scale

Supervisors Form a Ring

Configuration

• Config files stored in ./config• Default values provided by

• Scaffolding

• default.toml

• Override defaults through• ENV variable

• user.toml

• Over the network

What do applications need to run?

Dynamic configuration

Service group level

Uses the ring

Supervisors Support Clustering Topologies

Self Organizing Applications

• Habitat applications can self organize

• Typical container pattern• Bake intent into the container• Redis-master container image• Redis-follower container image• Introduces image sprawl

Self Organizing Applications

•Habitat pattern• One container image• Launch images with knowledge of peers• Application self organizes

External Actors

Health and Status

Supervisor Debugging

Supervisors provide a REST API

What Habitat brings to Containers:

• Build containers from the Application down• Build the minimal viable container by default• Export containers in a variety of deployable formats• Automatically export containers with:

• Service Discovery• Configuration Management• Supervisor API• Clustering Topology Support

What about those Developer Services?

Habitat and Developer Services

● Define How Software is Built○ Language Agnostic

● Define How Software is Deployed○ Export software to a variety of deployable formats (like containers)

● Define How Software is Run○ Common management plane for any software○ Common protocol for

■ Config■ Cluster

○ Common API for monitoring

Habitat and Developer Services

Quickly and easily build, deploy, and run any software across a variety of compute runtimes.

Habitat is automation for Application Runtimes & Application Services to deliver Cloud Native Developer Services

It’s all open source

Apache License

Questions

• Join the Habitat Slack Team - http://slack.habitat.sh/ • Work through the tutorial at https://www.habitat.sh/tutorials/• Explore Habitat packages on the depot - https://app.habitat.sh/• Explore the Habitat projects - https://github.com/habitat-sh • Read Habitat Blog posts - https://blog.chef.io/?s=habitat • Join the Habitat Forums - https://forums.habitat.sh/