1

Saltstack at Scale:Automating the Automation

Steven Gonzales | EMC | Principal Software Engineer

#SaltConf15

2

• We build and run BIG clouds

• Object Storage

• Remote teams

• Smart people

• SRE/DEV

• We are hiring!

EMC2

CLOUD SERVICES

3

• Exabyte+ Object Storage

• Datacenters around the world

• 10,000+ nodes

• 14+ Salt Masters

SALT AT SCALEWHAT DO WE MEAN BY SCALE?

4

STARTING OUTFIRST FLIGHT WITH SALT

5

• Salt Version 0.17.x

• Tool for remote command execution

• Developed use of states

• Monolithic Salt repo

• Used in only one project

STARTING OUTFIRST FLIGHT WITH SALT

6

• Poor code reuse across projects

• Divergent copies across projects

• Difficult to deploy pillar/states

• Hard to test

• Lack of consistent patterns

STARTING OUTFIRST FLIGHT WITH SALT

7

• Version control

• Design patterns

• Code reuse

• Testing

• Deployment

• Build system!

INFRASTRUCTURE AS CODETREAT IT LIKE A SOFTWARE PRODUCT

8

ENTER THE

FORMULA

PATTERNS & PRACTICES

9

• Promotes code reuse

• End divergence

• Project repos should only contain pillar data

• Test states independently

• Still needs standards

SALT FORMULAS WHY FORMULAS?

10

SALT FORMULALAYOUT

• _modules and _states

• Debian packaging

• The states folder (i.e.: docker)

• Map.jinja

• Test Directory

• Makefile

• Cookiecutter!

11

• Map.jinja

• Every setting is defined and has a default value

• OS map, defaults, and pillar merged into settings dict

• Pillar acts as an override

• One place to see configuration options

SALT FORMULAS SETTINGS & CONFIGURATION

12

SALT FORMULASETTINGS & CONFIGURATION

• States and templates access all data though the settings dict

13

• Provision VMs

• Built-in Salt provisioner

• Fast for local testing

• Disposable environment

• Private Networks

• Vagrant up!

SALT FORMULASTESTING WITH VAGRANT

14

• Write functional tests in python

• Nosetests

• Tox

• Execute against the Vagrant environment

SALT FORMULASPYTHON TEST TOOLS

15

• Install to /srv/formula

• Support installing _modules and _states

• Support versioning & dependencies

• Easy rollback

• Not every network has access to Git

SALT FORMULAS HOW TO INSTALL?

16

• Don’t reinvent the wheel!

• `apt-get install salt-formula`

SALT FORMULAS INSTALL WITH SYSTEM PACKAGES

17

AUTOMATING THE

AUTOMATIONBUILD AND DEPLOYMENT

18

• Salt-formula

• Bootstraps salt masters/minions

• Installs formulas/pillar from system pkgs

• Salt masters highstate themselves

SALT INCEPTIONCORE TO AUTOMATION

19

• Formulas have devel and master branches

• Developers PR against devel

• Merged to master after successful build/testing

WORKFLOWSGIT BRANCHING

20

• Build server Go.cd

• Dashboard of build status

• Pipeline Templates

• Pipelines for devel and master

• Notifications

WORKFLOWSAUTOMATING THE BUILD

21

• Vagrant up & execute functional tests

• Build a package

• Push package into repo

• Devel to testing repo, master to prod repo

WORKFLOWSBUILD STEPS

22

WORKFLOWSFORMULA DEPENDENCIES

23

• Developers PR to merge to devel

• Build, package, & push to test repo

• Auto-install to test environment

• Merge devel to master

• Build, package, & push to prod repo

• Ready for Production

WORKFLOWTHE FULL PIPELINE

24

• Still plenty to convert to formulas

• Tests for pillar data

• Improve tooling for formula dependencies

• Expanded use of external pillar

NEXT STEPSALWAYS ROOM TO IMPROVE

25

Thoughtworks Go.cd:

Continuous build and delivery system

http://www.go.cd/

Vagrant:

Lightweight VM provisioning software

https://www.vagrantup.com/

Tox Automation Project:

Python test automation for running tests inside a virtual environment

https://tox.readthedocs.org

USEFUL LINKS

Nosetests:

Python Test Runner

https://nose.readthedocs.org

Cookiecutter:

Project for rendering new projects from a jinja template

https://cookiecutter.readthedocs.org