microservices manchester: security, microservces and vault by nicki watt

Security, Microservices

& Vault

Nicki Watt @techiewatt

1

http://www.microservicesmanchester.com

http://www.microservicesmanchester.com

About Me

• Hands on Lead consultant at OpenCredo

• Co-author Neo4j In Action

• Twitter: @techiewatt

2

Agenda

• Introduction • Framework for assessing challenges • Vault • Conclusion

3

4

Introduction

5

You’ve already heard the stories of how …

6

from the monolith …image credit: http://lovealwaysbear.blogspot.co.uk/2011_01_01_archive.html

Applications

http://lovealwaysbear.blogspot.co.uk/2011_01_01_archive.html

7

to microservices

image credit: http://www.guinnessworldrecords.com/world-records/most-tennis-balls-held-in-the-mouth-dog

Applications

http://www.guinnessworldrecords.com/world-records/most-tennis-balls-held-in-the-mouth-dog

8

to microservices

image credit: http://www.guinnessworldrecords.com/world-records/most-tennis-balls-held-in-the-mouth-dog

Not ev

ery pr

oblem

needs

microser

vices!

Applications

http://www.guinnessworldrecords.com/world-records/most-tennis-balls-held-in-the-mouth-dog

9

from Silo’d teams with manual release processes

image credit: http://kittypluscoco.blogspot.co.uk/2011/04/day-at-dog-park.html

Teams

http://kittypluscoco.blogspot.co.uk/2011/04/day-at-dog-park.html

10

image credit: http://www.notey.com/@coolshitibuy/external/10054533/ruffwear-approach-dog-backpack.html

to agile teams with fast, automated software delivery

DevOps!

Teams

http://www.notey.com/@coolshitibuy/external/10054533/ruffwear-approach-dog-backpack.html

11

But …

12

What do you mean “It’s going live today” ?

image credit: https://www.facebook.com/EarltheGrump/photos

Security ?

https://www.facebook.com/EarltheGrump/photos

13

image credit: https://www.facebook.com/EarltheGrump/photos

SECURITY BOLTED ON AT THE END!

#FAIL!

Security ?

What do you mean “It’s going live today” ?

https://www.facebook.com/EarltheGrump/photos

15

image credit: http://www.beauswish.org/wp-content/uploads/2016/04/arianna.jpg

DevSecOps!

agile teams (with security as a 1st class citizen) practicing fast, secure,

automated software delivery

http://www.beauswish.org/wp-content/uploads/2016/04/arianna.jpg

Delivery Pipeline

17

http://www.devsecops.org/blog/2016/5/20/-security

<— Shifting Security to the Left Shannon Lietz

DEV

TEST

OPS

SECURITY

http://www.devsecops.org/blog/2016/5/20/-security

“secure reasoning” should be

in the forefront of every engineers minds

18

Microservice example:

Big retail store selling goods which includes a typical “web store”

19

20

user service

product service

Example: web store

21

user service

product service

Example: web store

external system XXX

22

user service

product service

Example: web store

external system XXX

sensitive data

passwords, keys

23

Example: web store

external system XXX

store api

store front

user service

product service

sensitive data

passwords, keys

24

sensitive data

store api

store front

user service

product service

external system XXX

passwords, keys

Example: web store

Where do we start ?

25

Know thy playground!

• What infrastructure? • What tech stacks? • What databases? • What type of delivery channels?

26

27

sensitive data

store api

store front

user service

product service

external system XXX

passwords, keys

Example: web store

28

sensitive data

store api

store front

user service

product service

external system XXX

passwords, keys

Example: web store

29

sensitive data

store api

store front

user service

product service

external system XXX

passwords, keys

Example: web store

30

A framework for thinking about

security …

31

NIST Cyber Security Framework

32

NIST Cyber Security Framework

33

IDENTIFY

PROTECT

DETECT

RESPOND

RECOVER

What stuff needs protecting?

What can I do to protect it?

How will I know if bad stuff happens?

What should I do when bad stuff happens?

How can I get my system back up and running after bad stuff has happened?

34

IDENTIFY What stuff needs protecting?

35


Threat Modelling

36


Attack Trees https://www.schneier.com/academic/archives/1999/12/attack_trees.html

https://www.schneier.com/academic/archives/1999/12/attack_trees.html

38

IDENTIFY

sensitive data

external system XXX

store api

store front

passwords, keys

user service

product service

steal sensitive user

data

store api

store front

sensitive data

passwords, keys

user service

product service

external system XXX39

IDENTIFY

gain access to internal network


data

attack store front

/ API

sniff non encrypted

traffic

SQL Injection

Alter query to get data

modify data in DB

social engineering

sniff non encrypted

traffic

external system XXX

sensitive data

passwords, keys

user service

product service

40

IDENTIFY

store api

store front

attack store front

/ API

sniff non encrypted

traffic

SQL Injection



data

modify data in DB


IDENTIFY

store api

store front

sensitive data

passwords, keys

user service

product service



data

social engineering

sniff non encrypted

traffic

store api

store front

sensitive data

passwords, keys

user service

product service


IDENTIFY



data

attack store front

/ API

sniff non encrypted

traffic

SQL Injection


modify data in DB

social engineering

sniff non encrypted

traffic

Security, and actually being able to do things,

always requires a trade off!43

store api

store front

sensitive data

passwords, keys

user service

product service


PROTECT

attack store front

/ API

sniff non encrypted

traffic

SQL Injection


modify data in DB

HTTPS Use prepared statements

build web app vuln verification into CI/CD


social engineering

sniff non encrypted

traffic


data

HTTPS

Firewall

store api

store front

sensitive data

passwords, keys

user service

product service


PROTECT

attack store front

/ API

sniff non encrypted

traffic

SQL Injection


modify data in DB




social engineering

sniff non encrypted

traffic


data

HTTPS

Firewall

store api

store front

sensitive data

passwords, keys

user service

product service


PROTECT

attack store front

/ API

sniff non encrypted

traffic

SQL Injection


modify data in DB




social engineering

sniff non encrypted

traffic


data

HTTPS

Firewall

cfssl

store api

store front

sensitive data

passwords, keys

user service

product service


PROTECT

attack store front

/ API

sniff non encrypted

traffic

SQL Injection


modify data in DB




social engineering

sniff non encrypted

traffic


data

HTTPS

Firewall

store api

store front

sensitive data

passwords, keys

user service

product service


DETECT

Log suspicious queries

Log HTTP requests

Log HTTP requests

attack store front

/ API

sniff non encrypted

traffic

SQL Injection


modify data in DB




social engineering

sniff non encrypted

traffic


data

HTTPS

Firewall

IDS

store api

store front

sensitive data

passwords, keys

user service

product service



infect employee computer

install malware via

email

sniff non encrypted

traffic

compromise user data

attack store front

/ API

sniff non encrypted

traffic

SQL Injection


modify data in DB

HTTPS

HTTPS

Firewall

antivirus

Use prepared statements

IDS


Log HTTP requests

Log HTTP requests


DETECT

Distributed logging

capability

Container level

loggingAlerting

capability

Infrastructure level

logging

Serverless logging

???

store api

store front

sensitive data

passwords, keys

user service

product service


RESPOND

Redirect to HTTPS

Block consistent offenders

Adjust firewall rules Block attackers


Log HTTP requests

Log HTTP requests

attack store front

/ API

sniff non encrypted

traffic

SQL Injection


modify data in DB




social engineering

sniff non encrypted

traffic


data

HTTPS

Firewall

IDS

Change DB Password Reset users passwords

Inform users

Redirect to HTTPS

store api

store front

sensitive data

passwords, keys

user service

product service




RECOVER

Redirect to HTTPS


Adjust firewall rules Block attackers


Log HTTP requests

Log HTTP requests

attack store front

/ API

sniff non encrypted

traffic

SQL Injection


modify data in DB




social engineering

sniff non encrypted

traffic


data

HTTPS

Firewall

IDS

Change DB Password Reset users passwords

Inform users

Redirect to HTTPS

Restore from backup

Fix Code, Blue/Green deploys:

redeploy microservice(s) redeploy infrastructure

54

RECOVER

Trash & burn! is your friend

• Due diligence: know thy playground

• Think holistically: identify, protect, detect, respond, recover

Summary

55

Make security a 1st class citizen

in your thinking process!

• Multiple, diverse, interconnected services

• More varied attack surfaces

• Harder to track what’s going on (distributed, multi facetted logging capabilities)

• Transient components

• Dynamic transport level encryption (HTTPS)

• Authentication & Authorisation (see David’s talk :)

• Trash & burn recovery strategies

Microservice security challenges

56

Onto the practical bit …

58

59

A tool for managing secrets and other sensitive content

60

Deployment Tools

Application Component / Microservices

service 1 service 2

Human Users

61

• Unified API to access multiple backends • ACL policies - who can access what • Audit Logs

62

UnsealInit

service 1

service 2

Allow token to be used by tools to access secrets

Acquire policy constrained

token

Create microservice mount or area, add

secrets

System X

63

$ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b

Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again.

Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed.

$ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4baSealed: trueKey Shares: 3Key Threshold: 2Unseal Progress: 1

Vault init & unseal

$ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91Sealed: trueKey Shares: 3Key Threshold: 2Unseal Progress: 0

64





Vault init & unseal


65





Vault init & unseal


66

Success! Ready for use

67

Unseal

Create segregated area, policies, add secrets

Init


tokenAllow token

to be used by tools to access secrets

service 1

service 2

System X

68

$ vault mount -path=usersvc generic Successfully mounted 'generic' at ‘usersvc'!

$ vault mounts Path Type Default TTL Max TTL Description cubbyhole/ cubbyhole n/a n/a per-token private secr ... secret/ generic system system generic secret storage sys/ system n/a n/a system endpoints used f... usersvc/ generic system system

Vault create new mount

69

$ vault write usersvc/db-password value=ASDKJ234SF*2 Success! Data written to: usersvc/db-password

$ vault read usersvc/db-password Key Value lease_duration 2592000 value ASDKJ234SF*2

Vault write, then read back secret

70

$ cat usersvc.policy path "usersvc/*" { policy = "read" }

$ vault policy-write usersvc usersvc.policy Policy 'usersvc' written.

Vault create custom policy

71

Unseal

Allow token to be used by tools to access secrets

Init


token

service 1

service 2

Create segregated area, add secrets

System X

72

Basics of Vault complete!

Getting sensitive data into microservices …

73

74

# Embedded Configspring.datasource.url=jdbc:mysql://localhost/testspring.datasource.username=dbuserspring.datasource.password=dbpassspring.datasource.driver-class-name= com.mysql.jdbc.Driver

Java Code@Componentpublic class MyBean {

private final JdbcTemplate jdbcTemplate;

@Autowired public MyBean(JdbcTemplate jdbcTemplate) { this.jdbcTemplate = jdbcTemplate; }

// ...

}

Starting point …

user service

db1

75





// ...

}

Starting point …

user service

db1

Separa

te Code

and Co

nfig -

Especi

ally Se

crets!!

76





// ...

}

Starting point …

user service

db1

Separa

te Code

and Co

nfig -

Especi

ally Se

crets!!DETECT

https://github.com/michenriksen/gitrob

https://github.com/awslabs/git-secrets

https://github.com/michenriksen/gitrob

https://github.com/awslabs/git-secrets

77

Options

• Push secrets in

• Pull secrets out

• Variations of the above …

78

Push secrets in …

user service

db1

1authenticate

2orchestration / deployment platform

3provide value as environment variables

read secret/db-password

79

user service

db1

1authenticate

2read secret/db-passwordorchestration /

deployment platform


$ vault auth e2d0a065-xxxx-yyyy-zzzz Successfully authenticated! You are… token_policies: [default, usersvc]

$ vault read usersvc/db-password Key Value --- ----- refresh_interval 2592000 value MyClearTextPassword

1

2

80

user service

db1

1authenticate

2read secret/db1orchestration /

deployment platform


$ # Start docker container,pass in vars docker run --name usersvc -e DB_USER="MyDBName" -e DB_PASSWORD="MyClearTextPassword" -d usersvc:v1

3

81

Steal Sensitive User DataIDENTIFY


data


data


gain access to user DB

gain access to running user

microservice(s)

dump startup config

steal plaintext password

social engineering

the-machine$ docker ps

CONTAINER ID IMAGE ... CREATED STATUS NAMES 9950ea8e3c59 product-service:v1 ... 4 days ago Up 4 days prodsvc 29b9ebca6dab user-service:v2 ... 5 days ago Up 5 days usersvc

82


microservice(s)

83


find a disgruntled employee

dump startup config

the-machine$ docker inspect 29b9ebca6dab

[ { "Id": “29b9ebca6dab147991201a5c61b72d3d546885ca8fd…”, "Created": "2016-06-27T21:26:16.126414991Z", "Args": [ "-jar", "UserService" ], "Config": { "Hostname": "29b9ebca6dab", "Env": [ “DB_USER=MyUserName", “DB_PASSWORD=MyClearTextPassword", “VAR1=something-else“ ], "Cmd": [ "java", "-jar", "UserService" ], ... } ]

84




[ { "Id": “29b9ebca6dab147991201a5c61b72d3d546885ca8fd…”, "Created": "2016-06-27T21:26:16.126414991Z", "Args": [ "-jar", "UserService" ], "Config": { "Hostname": "29b9ebca6dab", "Env": [ “DB_USER=MyUserName", “DB_PASSWORD=MyClearTextPassword", “VAR1=something-else“ ], "Cmd": [ "java", "-jar", "UserService" ], ... } ]


85




microservice(s)

dump startup config

social engineering

PROTECT

don’t expose as plain text


data


limit user access

Vault Response Wrapping

86

Push secrets in … (take 2)

87

Push secrets in …

user service

db1

1authenticate



read secret/db-password

87

user service

db1

1authenticate

2read wrapped secretorchestration /

deployment platform

3provide wrapped value as environment variables

4

unwrap

Push wrapped secrets in …

88

user service

db1

1authenticate


deployment platform


4

unwrap

$ vault read -wrap-ttl=60s usersvc/db-password Key Value --- ----- wrapping_token: 57ccef32-471d-869 wrapping_token_ttl: 60 wrapping_token_creation_time: 2016-06-28 22:..

2

89

user service

db1

1authenticate


deployment platform


4

unwrap

$ # Start docker container,pass in vars docker run --name usersvc -e DB_USER="MyDBName" -e DB_PASSWORD="57ccef32-471d-869" -d usersvc:v1

3

90

user service

db1

1authenticate


deployment platform


4

unwrap

$ vault unwrap 57ccef32-471d-869 Key Value --- ----- refresh_interval 2592000 value MyClearTextPassword

4

91

dump startup config


[ { "Id": “29b9ebca6dab147991201a5c61b72d3d546885ca8fd…”, "Created": "2016-06-27T21:26:16.126414991Z", "Args": [ "-jar", "UserService" ], "Config": { "Hostname": "29b9ebca6dab", "Env": [ “DB_USER=MyUserName", “DB_PASSWORD=57ccef32-471d-869", “VAR1=something-else“ ], "Cmd": [ "java", "-jar", "UserService" ], ... } ]

92



microservice(s)

dump startup config


PROTECT


data




limit user access

93



microservice(s)

dump startup config


PROTECT


data




limit user access

94



microservice(s)

dump startup config


PROTECT


data


gain access to user DB steal wrapped

password

get real password

limit user access

95

user service

db1

1authenticate


deployment platform


4

unwrap

$ vault unwrap 57ccef32-471d-869 error reading cubbyhole/response: Error making API request.

URL: GET https://vault:8200/v1/cubbyhole/response Code: 400. Errors:

* permission denied

4

96



microservice(s)

dump startup config


PROTECT


data

DETECT



password

get real password

Raise TOFU alarm

Audit access

limit user access

97



microservice(s)

dump startup config


PROTECT


data

DETECT


RESPOND


password

get real password

Raise TOFU alarm

Audit access

change DB password

limit user access

98



microservice(s)

dump startup config


PROTECT


data

DETECT


RESPOND


password

get real password

Raise TOFU alarm

Audit access

change DB password

Expect secrets to change. Make a habit of changing them regularly.

It will naturally force you to put measures in place. limit user access

• Dynamic Secrets: Auto generate credentials on the fly

Other handy options

99

100

user service

db1

1authenticate

2read dynamic passwordorchestration /

deployment platform


0

Human / Other System

Users

101

user service

db1

1authenticate



$ vault mount postgresql Successfully mounted 'postgresql' at 'postgresql'!

$ vault write postgresql/config/connection connection_url="postgresql://vault:somepassword@yourhost:5432/postgres"

$ vault write postgresql/roles/usersvc-ro \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD ‘{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA users TO \"{{name}}\";" Success! Data written to: postgresql/roles/

read dynamic password


Users

0

0

102

user service

db1

1authenticate








Users

0

0

103

user service

db1

1authenticate








Users

0

0

104

user service

db1

1authenticate



$ vault read postgresql/creds/usersvc-ro Key Value lease_id postgresql/creds/usersvc-ro/c888a097-b0e2-26a8-b306-fc7c84b98f07 lease_duration 3600 password 34205e88-0de1-68b7… username vault-14301-usersvc-ro



Users

0

2

105

user service

db1

1authenticate



$ # Start docker container,pass in vars docker run --name usersvc -e DB_USER="vault-14301-usersvc-ro" -e DB_PASSWORD="34205e88-0de1-68b7" -d usersvc:v1


• Dynamic Secrets: Auto generate creds on the fly

• Ability to combine security primitives dynamic secrets + resource wrapping

Other handy options

106

107




microservice(s)

dump startup config


PROTECT


data

DETECT

steal wrapped password


get real password

Raise TOFU alarm

Audit access

RESPOND

change DB password

108




microservice(s)

dump startup config


PROTECT


data

DETECT



get real password

Raise TOFU alarm

Audit access

RESPOND

change DB password

use time limited dynamic creds

109




microservice(s)

dump startup config


PROTECT


data

DETECT



get real password

Raise TOFU alarm

Audit access

RESPOND

change DB password

use time limited dynamic creds

compromise orchestration

platform

Turtles all the way down!

111




microservice(s)

dump startup config

compromise orchestration

platform



data

steal vault token

get db password

1

2

34

Defense in Depth

Put enough hurdles in the way of attackers for you to stop

when you can, but if not, to be able to …

- realise what’s going on

- react before too much damage is done

112

• Centralised Secrets Management • API - helps with automation • Tries to address concerns across full

security lifecycle • But still very new & maturing

Vault Summary

113

• Encryption as a service: offload responsibility to Vault

• PKI: Generates X.509 certificates dynamically based on configured roles

• SSH: Dynamically generates SSH credentials for remote hosts

Other Handy Features

114

Conclusion

115

116

Make security a first class citizen!

Don’t try and just bolt it on at the end!

117

Think holistically about security

Don’t stop at the protect stage!

118

Choose the right tech for the job

Microservice architectures add complexity

119

Do your best!but don’t do nothing!

Questions? Nicki Watt

@techiewatt

120

microservices manchester: security, microservces and vault by nicki watt

Technology