Puppet Camp April 9th 2013

What’s in a name?

Andrew Fong and Gary Josack fong@dropbox.com gary@dropbox.com

About Dropbox

•  Thousands of instances/servers

• Mostly Python Stack

•  EC2 and Large Self Datacenters

• Over a billion file syncs per day

•  Thousands of MySQL Shards

•  4 SREs and 1 DBA

A story of a startup...

Hostapuppet.com

Chapter One

•  1 or 2 teams

•  Couple of hosts

• Webserver and a database

• Maybe one ops guy

What Ops People Like

•  Simplicity

•  Repeatability

•  Assurances that things remain consistent

•  Puppet / Configuration management

Config Management

node 'www1.example.com' { include common include apache include squid } node 'db1.example.com' { include common include mysql }

My First Puppet Config

Node ‘mickey.hostapuppet.com’ { include common include webserver include sudoers

} Node ‘donald.hostapuppet.com’{

include common include db include sudoers

}

Sudoers Module

… file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", } …

Sudoers File itself

Host_Alias DONALD=donald.hostapuppet.com

Host_Alias MICKEY=mickey.hostapuppet.com

db_guy DONALD=(all) NOPASSWD: ALL

ops_guy MICKEY=(all) NOPASSWD: ALL

Chapter Two: A growing service

•  A few teams

•  2 or 3 services

• multiple types of hosts

–  Web

–  API

–  DB

Hostnames

•  sjc-web[1-N]

•  sjc-db[1-N]

•  sjc-api[1-N]

Host Regex

$hosttype = inline_template('<%= hostname.sub(/\w+-([a-z]+)\d*/){$1} %>’)

Hosttypes $hosttype = inline_template('<%= hostname.sub(/\w+-([a-z]+)\d*/){$1} %>’)

if $hosttype == ‘web’ { include sudoers include web

} If $hosttype == ‘db’ {

include sudoers include db

}

if $hosttype == ‘api’ { include sudoers include api

}

Back to sudoers

… file { "/etc/sudoers": owner => root, group => root, mode => "440", source => "puppet:///modules/sudo/sudoers", } …

Sudoers File itself

Host_Alias WEB=sjc-web*

Host_Alias DB=sjc-db*

Host_Alias API=sjc-api*

database_guy DB=(all) NOPASSWD: ALL

ops_guy WEB=(all) NOPASSWD: ALL

api_team API=(all) NOPASSWD: ALL

Hypergrowth

0

20

40

60

80

100

120

0 2 3 5

Users (millions)

Users (millions)

Chapter 3: An Expanding Infrastructure

•  Lots of new hires!

•  A bunch more developers

•  Some PMs

•  Some Designers

All Kinds Of Problems…

•  Boxes of same hardware class running

different services

•  Boxes serving more than one role

(remember sudoers?)

• Deploying or moving hosts quickly

Renaming a host

• Update dns

• Update dhcpd.conf

•  Push both

• Update puppet configs

• Update code

OMG I JUST RENAME HOSTS!

Sudoers File From Chapter Two…

Host_Alias WEB= sjc-web* Host_Alias API=sjc-api* Host_Alias DB=sjc-db* database_guy DB=(all) NOPASSWD: ALL ops_guy WEB=(all) NOPASSWD: ALL api_team API(all) NOPASSWD: ALL

Sudoers File in Chapter 3

Host_Alias WEB= sjc-web* Host_Alias API=sjc-api*,sjc-web550,sjc-web551,sjc-web552,sjc-web553 Host_Alias DB=sjc-db* database_guy DB=(all) NOPASSWD: ALL ops_guy WEB=(all) NOPASSWD: ALL api_team API(all) NOPASSWD: ALL

Dropbox

• We did all that.

• We’re still paying the taxes for doing

that.

•  But there is a light at the end of the

tunnel…

ABSTRACT THE SERVICE

FROM THE HOST!

So what does that mean?

• Make hosts role agnostic

• Do not require invasive changes

• Simple interfaces

Making hosts role agnostic

• Positional

• Serial Numbers

• Anything that doesn't change

The Dropbox Plan

• Positional names

• Custom Machine Database

• External Node Classifier

• Transitioning Puppet configs

• Naming service(s) for convenient names

Service/Machine Management Database

• Universal Source of Truth

• Manage roles / attributes

• Generated configs

- Gmond, Nagios, etc

What exactly is the ENC

• External Node Classifier

• Inject variables (and other) from external

process

• YAML Output

Part 2: External Node Encoders

Sudoers++

• Move from monolithic to modular

• Includes! (Weird caveats)

• Just use ALL for Host_Lists

Sudoers at Dropbox

Part 3: Helper Functions

Sudoers with tags

Sudoers with tags

Provisioning

• Preload MDB, DNS, DHCPD, etc.

- Set it and forget it

• Have spares ready for any roles

• Assigning a role is one command

• No more renames!

Dynamic Naming w/ PowerDNS

Dynamic Naming w/ PowerDNS

Zookeeper

• ZKNS included with the Vitess project

• ZK is in use at various different companies (YouTube, Twitter, AirBnB)

Q&A

λ FAQ #1: Are you hiring? - Yes! Come talk to us. :)