puppetconf 2016: puppet troubleshooting – thomas uphill, wells fargo

https://goo.gl/8LyZzN

puppet troubleshootingThomas Uphill<[email protected]>


Mastering Puppet / Puppet CookbookPackt Pub


LOPSA / SASAG / PUGS

When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth

Sir Arthur Conan Doyle

4


Troubleshooting 101

● document current state● discover recent changes

○ audit everything, even things you don't touch○ never assume it's magic

● change one thing at a time○ if it doesn't fix the problem, revert it

5

1 une uno um unis odin jeden jedward yksi 一 एक واحد אחד


Troubleshooting 101

● don't be afraid of diving deep○ you can't fix what you don't understand

● use low level tools

6


TAKE HOME

puppet is an HTTPS service

7

problemsconnectioncertificatecatalog compilationcatalog applicationscopepry

8


can't find puppet

$ puppet config print serveryourface$ puppet config print confdir/home/thomas/.puppetlabs/etc/puppet

# puppet config print serverpuppet# puppet config print ca_serverpuppet# puppet config print confdir/etc/puppetlabs/puppet

9

puppet agent Runs as root…puppetserver Runs as puppet.

10

Basic UNIX permissions


11


can't find puppet

dns

files/etc/hosts

nsswitch.conf# host puppetHost puppet not found: 3(NXDOMAIN)# nslookup puppetServer: 10.0.2.3Address: 10.0.2.3#53

** server can't find puppet: NXDOMAIN# dig puppet

# getent hosts puppet127.0.0.1 puppet.example.com puppet localhost localhost.localdomain localhost4 localhost4.localdomain4

# ping puppetPING puppet.example.com (127.0.0.1) 56(84) bytes of data.64 bytes from puppet.example.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.035 ms

12


can't connect to puppet

node puppetserver

networkpixies

8140

server

masterport

13



Verify you can connect:

# nc -v puppet.example.com 8140

Ncat: Version 7.12 ( https://nmap.org/ncat )

Ncat: Connected to 192.168.1.1:8140

14



Q: How can you know you are talking to puppet?

A: Use the REST API

15



$ curl -k

https://puppet.example.com:8140/puppet-ca/v1/certificate/ca?environment=production

-----BEGIN CERTIFICATE-----

MIIFhDCCA2ygAwIBAgIBATANBgkqhkiG9w0BAQsFADAoMSYwJAYDVQQDDB1QdXBw

ZXQgQ0E6IHB1cHBldC5leGFtcGxlLmNvbTAeFw0xNjEwMDgyMDEyNDRaFw0yMTEw

…qsHOzPEQwJS5q1yPciBQ5jsKCB5vXJEdnPMhA5GBx9rp91hBsiWURlyWt0DMqMTq

VqcUEGNc267LrCzKJsFBHCNO53e8z2I0

-----END CERTIFICATE-----

masterportca_serverpuppet orpuppet-ca api endpoint key environment

https://docs.puppet.com/puppet/latest/reference/http_api/http_api_index.html

16

https://www.google.com/url?q=https://docs.puppet.com/puppet/latest/reference/http_api/http_api_index.html&sa=D&ust=1476991027648000&usg=AFQjCNFlzAY_Geo_nwfE2Tp_yFsNxdkM_w

https://www.google.com/url?q=https://docs.puppet.com/puppet/latest/reference/http_api/http_api_index.html&sa=D&ust=1476991027649000&usg=AFQjCNFm3b95h2i1jXprMUUrrJCZnbuxug



$ gnutls-cli --port 8140 puppet.example.com

--x509cafile=ca.pem

$ openssl s_client -connect puppet.example.com:8140

GET /path/to/thing/you/want

Accept: PSON/YAML/Text/???

puppetmaster

(puppetserver)nodeHTTPS/8140

17

lsof -i :8140mtrtracerouteping


in order to authenticate...you must first authenticate.

18



$ curl --cert certs/cottage.pem --key private_keys/cottage.pem.orig --cacert

certs/ca.pem https://puppet.example.com:8140/puppet/v3/environments | jq

… "environments": {

"production": {

"settings": {

"modulepath": [

"/etc/puppetlabs/code/environments/production/modules",

"/etc/puppetlabs/code/modules",

"/opt/puppetlabs/puppet/modules"

],

"manifest": "/etc/puppetlabs/code/environments/production/manifests",

"environment_timeout": 0,

"config_version": ""

19


API FTW

$ curl --cacert certs/ca.pem \

--cert certs/cottage.pem \

--key private_keys/cottage.pem \

-H 'Accept: pson' \

https://puppet.example.com:8140/puppet/v3/catalog/cottage?environment=production \

> cottage.pson

20

catalog


PSON

Puppet JSON → PSON$ jq <pson>

$ curl … | jq

JQ? sed for JSON

$ jq .environment catalog.pson

$ jq .classes catalog.pson

$ jq '.resources | length' catalog.pson

$ jq '.resources[] | select(.type == "Stage")' catalog.pson

21


PSON

Puppet JSON → PSON$ jq <pson>

$ curl … | jq

JQ? sed for JSON

$ jq .environment catalog.pson

$ jq .classes catalog.pson

$ jq '.resources | length' catalog.pson

$ jq '.resources[] | select(.type == "Stage")' catalog.pson

Sometimes it's good to go...

22

problemsconnection

certificatecatalog compilationcatalog applicationscopepry

23


can't get certificate

already signed (clean) dates off - expired CA, expired cert

openssl x509 -in cert.pem -text

puppet cert fingerprint host.example.compuppet cert print host.example.compuppet cert clean host.example.com

ntpq -pchronyc sources

rm /etc/puppetlabs/puppet/ssl/*/hostname*

24

Intermission

25

x.509the crashcourse


x.509

PKI

CA

SSL

TLS

CRL

REQ

RSA

DSA

DH

PKCS7

PKCS8

PKCS12

MD2

MD5

SHA1

SHA256

SHA384

SHA512

PEM

DER

OID

RFC

KS

TS

BOB

ALICE

"ENCRYPTION""ENCRYPTION"

26


x.509

PKI

CA

SSL

TLS

CRL

REQ

RSA

DSA

DH

PKCS7

PKCS8

PKCS12

MD2

MD5

SHA1

SHA256

SHA384

SHA512

PEM

DER

OID

RFC

KS

TS

BOB

ALICE

27


x.509

Public Key Infrastructure

Certificate Authority

Trust

https://www.flickr.com/photos/genvessel/431100596

28


ca.pem

x.509puppetmaster

CERTIFICATE

node

CA?

ca.pem

serverca_server

29


node.crtca.crt

x.509puppetmaster

CERTIFICATE

node

node.csr

node.crt

serverca_server

30

cert me pls?


puppet.pemca.pem

x.509puppetmaster

CERTIFICATE

node

serverca_server

31


OpenSSL

View Certificate:

openssl x509 -in ca.pem -text -noout |lessValidityX509v3 Key Usage

View Fingerprint:

openssl x509 -in ca.pem -fingerprint -noout

openssl x509 -in ca.pem -fingerprint -noout -sha256

SHA1 (default)

Verify Certificate:

openssl verify -CAfile ca.pem puppet.pem

32


OpenSSL (Puppet shortcuts)

View Certificate:

puppet cert print ca

View Fingerprint:

puppet cert fingerprint ca

List Certificates:

puppet cert list -a

33


node.crtca.crt

CRLpuppetmaster

CERTIFICATE

node

crl?

crl

34

crl


CRL

# openssl crl -in crl.pem -text -noout |grep Serial

Serial Number: 02

# openssl x509 -in cert.pem -text -noout |grep Serial

Serial Number: 3 (0x3)

35


CRL# cat ca_crt.pem ca_crl.pem >combined.pem

# openssl verify -CAfile combined.pem -crl_check cottage.pem

cottage.pem: OK

# puppet cert clean cottage

Notice: Revoked certificate with serial 3

Notice: Removing file Puppet::SSL::Certificate cottage at 'puppet/ssl/ca/signed/cottage.pem'

Notice: Removing file Puppet::SSL::Certificate cottage at 'puppet/ssl/certs/cottage.pem'

# cat ca_crl.pem ca_crt.pem >combined.pem again?

# openssl verify -CAfile combined.pem -crl_check cottage.pem

cottage.pem: CN = cottage

error 23 at 0 depth lookup:certificate revoked

36


Modulus

n = pq

OpenSSL

# openssl rsa -noout -modulus -in ca_key.pem |sha256sum

69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38

# openssl x509 -noout -modulus -in ca_crt.pem |sha256sum


BIG *RSE PRIME

37


Modulus

n = pq

OpenSSL

# openssl rsa -noout -modulus -in ca_key.pem |sha256sum


# openssl x509 -noout -modulus -in ca_crt.pem |sha256sum


BIG *RSE PRIME

$ puppet agent -tError: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.Certificate fingerprint: D4:D3:76:F1:6B:51:83:3C:4B:72:69:BF:BC:B0:80:94:79:75:1A:3B:D8:29:F5:EF:81:2C:44:35:21:93:CE:FDTo fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certificate.On the master: puppet cert clean cottageOn the agent: 1a. On most platforms: find /home/thomas/.puppetlabs/etc/puppet/ssl -name cottage.pem -delete 1b. On Windows: del "\home\thomas\.puppetlabs\etc\puppet\ssl\certs\cottage.pem" /f 2. puppet agent -t

38

The certificate retrieved from the master does not match the agent's

private key.


OpenSSL recap

● x509view certificatecheck expirationcheck serial number

● crlrevoked cert serial#

● verifyverify cert with CA and/or CRL

● modulushow the cert was encrypted

39


Problem workers

node

puppetserver

puppetserver

loadbalancer

devel

problempuppetserver

problem

production

mod_proxy_balancer

40


Problem workers

puppetserver

--logdest /var/log/puppetlabs/puppetserver/problem.log--debug--profile

logrotate

41


problem/bugfixes environments

configure r10k to make environments automatically

environment.conf

- modulepath (debug module) ← what is that?

42


Compiling

$ sudo puppet master \ --compile problem.example.com \ --debug --trace \ --logdest /tmp/problem.puppet.log \ --environment sandbox

{ "data": { "resources": [ { "title": "main", "exported": false, "tags": ["stage"], "type": "Stage", "parameters": { "before": "Stage[post]",

JSON

Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'bucketdir': 'File[/var/lib/puppet/bucket]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/bucket"}'Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/lib/puppet/ssl/public_keys"}'Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'plugindest': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib"}'Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'fileserverconfig': 'File[/etc/puppet/fileserver.conf]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :backup=>false, :path=>"/etc/puppet/fileserver.conf"}'

43


Apply

$ sudo puppet apply \ --debug --trace \ --environment sandbox code.pp

$date = "+%S"exec {'epoch': command => "echo \$((`date $date` / 86400))", path => '/bin:/usr/bin:/sbin:/usr/sbin',}

Info: Applying configuration version '1415729233'Debug: Exec[epoch](provider=posix): Executing 'echo $((`date +%S` / 86400))'Debug: Executing 'echo $((`date +%S` / 86400))'Notice: /Stage[main]/Main/Exec[epoch]/returns: executed successfully

44

problemsconnectioncertificate

catalog compilationcatalog applicationscopepry

45


catalogs

46

catalog

fails to compile

duplicate resource modulepath/bad module

name

fails to apply

unpredictable exec bad/broken service bad/missing variable

47


fails to compile

48


help yourself

function verb() {

erb -P -x -T '-' $1 | ruby -c

is_ok $?

}

function vyaml() {

ruby -ryaml -e "YAML.load_file '$1'"

is_ok $?

}

function ppv() {

puppet parser validate $*

is_ok $?

}

49

function vepp() {

puppet epp validate $*

is_ok $?

}

function is_ok() { if [ $1 == 0 ]; then if [[ "$TERM" == *"xterm"* ]] || \ [[ "$TERM" == *"vt100"* ]]; then echo -e "\033[1mSyntax \033[32;1mOk!\033[0m" else echo "Syntax Ok!" fi else return $1 fi}


duplicate resource

separate into subclass ( package {'httpd'} )

virtual resources ( @user, @package, @service)

modulepath

puppet config print modulepath

root@puppet:~# puppet config print modulepath --environment production

/etc/puppet/environments/production/public:/etc/puppet/environments/production/modules

root@puppet:~# puppet config print modulepath --environment master

/etc/puppet/modules:/usr/share/puppet/modules

50

problemsconnectioncertificate

catalog compilation

catalog applicationscopepry

51


fails to apply

52


Unpredictable exec

#!/bin/bash

echo $JAVA_HOME /home/javadev/.bashrc

JAVA_HOME=/your/face

It works for me!

puppet runs aspuppet

53


Broken Service

service provider

hasstatus => true

/sbin/service $service status

/etc/init.d/$service status

/usr/bin/systemctl is-active $service

54


Bad/Missing Variable

$one = "1"

file {"pcone":

path => "/tmp/pc$one",

ensure => 'directory',

}

file {"pc1":

path => "/tmp/pc1",

ensure => 'file',

}

Info: Caching catalog for puppet.example.comError: Evaluation Error: Error while evaluating a Resource Statement, Cannot alias File[pc1] to ["/tmp/pc1"] at /root/pc.pp:6; resource ["File", "/tmp/pc1"] already declared at /root/pc.pp:2 at /root/pc.pp:6:3 on node puppet.example.com

55


Bad/Missing Variable

$PC = 'puppetconf'

pc {'one':

place => "/tmp/$PC",

type => "directory",

}

pc {'two':

place => "/tmp/$PC",

type => "file",

}

define pc ( String $place, String $type,) { file {"$title": path => $place, ensure => $type, }}

Info: Caching catalog for puppet.example.comError: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Cannot alias File[two] to ["/tmp/puppetconf"] at /root/define.pp:5; resource ["File", "/tmp/puppetconf"] already declared at /root/define.pp:5 at /root/define.pp:5:2 at /root/define.pp:15 on node puppet.example.com

56


Printing - Notify

notify {"$variable": }

57


chaining

notify {'something':

}->exec{'thingthatfails':

}->notify{'after': }

58


Checking

exec{'before resolv.conf':

command => '/usr/local/bin/puppet-debug before resolv.conf',

require => Class['debug']

} -> file { '/etc/resolv.conf':

source => template("dns/resolv.conf"),

noop => true,

}

class debug { file {'puppet-debug': path => '/usr/local/bin/puppet-debug', source => 'puppet:///modules/debug/puppet-debug', mode => 0755, }}

59


Debug Script… just an example#!/bin/bash

LOG=$(mktemp /tmp/puppet-debug.XXXXXX)

echo Puppet Debug -- $@ -- $(date) | tee $LOG

echo "-- Disk --" | tee -a $LOG

df -h |tee -a $LOG

df -i |tee -a $LOG

echo "-- Mem --" | tee -a $LOG

free | tee -a $LOG

echo "-- Files --" | tee -a $LOG

PUPPET=$(pgrep puppet)

for proc in $PUPPET

do

lsof -p $proc |tee -a $LOG

done

Puppet Debug -- before resolv.conf -- Fri Oct 24 01:13:34 EDT 2014-- Disk --Filesystem Size Used Avail Use% Mounted on/dev/mapper/VolGroup-lv_root 6.7G 2.5G 3.9G 39% /tmpfs 246M 0 246M 0% /dev/shm/dev/vda1 485M 80M 380M 18% /bootFilesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/VolGroup-lv_root 440640 79253 361387 18% /tmpfs 62783 1 62782 1% /dev/shm/dev/vda1 128016 50 127966 1% /boot-- Mem -- total used free shared buffers cachedMem: 502268 415488 86780 0 22176 172036-/+ buffers/cache: 221276 280992Swap: 835580 0 835580-- Files --COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpuppet 2058 root cwd DIR 253,0 4096 14 /rootpuppet 2058 root rtd DIR 253,0 4096 2 /puppet 2058 root txt REG 253,0 10600 36617 /usr/bin/rubypuppet 2058 root mem REG 253,0 156928 4134 /lib64/ld-2.12.sopuppet 2058 root mem REG 253,0 1926680 6282 /lib64/libc-2.12.so

60


Printing - Template

- scope.to_hash- reject a few- sort- print, one per line

file { "/tmp/puppet-debug.txt": content => inline_template("<% vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%= v %>\n<% end %>"), }

vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) };

vars.sort.each do |k,v| k=v\nend

61


Printing - Template_timestamp=2014-10-23 22:29:52 -0700architecture=x86_64augeasversion=1.0.0bios_release_date=01/01/2011bios_vendor=Bochsbios_version=Bochsblockdevice_vda_size=8589934592blockdevice_vda_vendor=6900blockdevices=vdacaller_module_name=clientcert=cookbook.example.comclientnoop=falseclientversion=3.7.1concat_basedir=/var/lib/puppet/concatdomain=example.comenvironment=productionfacterversion=2.2.0filesystems=ext4,iso9660fqdn=cookbook.example.comgid=roothardwareisa=x86_64hardwaremodel=x86_64hostname=cookbookid=rootinterfaces=eth0,lo

62

problemsconnectioncertificatecatalog compilationcatalog application

scopepry

63


Scope

The scene:

roles and profiles ntp server

class role::ntp { include ntp}

class ntp { include ntp::server}

64


Scope

The solution:

fully scope everything remember scope

class role::ntp { include ::ntp}

class ntp { include ntp::server}

65

problemsconnectioncertificatecatalog compilationcatalog applicationscope

pry

66


pry

IRB replacement

REPL

available at runtime

67

pryrepl.org

require 'pry'…# amazing code here# wow, much amaze… binding.pry…

https://www.google.com/url?q=http://pryrepl.org&sa=D&ust=1476991034197000&usg=AFQjCNHtuX-nwh57Hlk_CEaq5_qHbqbfag

https://www.google.com/url?q=http://pryrepl.org&sa=D&ust=1476991034197000&usg=AFQjCNHtuX-nwh57Hlk_CEaq5_qHbqbfag


pry demo

68

module Puppet::Parser::Functions newfunction(:pry) do |args| require 'pry' binding.pry endend

modules/pry/lib/puppet/parser/functions/pry.rb

node default { … pry() … }

manifests/site.pp


pry demo

#

69

#

…

From:

/etc/puppetlabs/code/environments/production/modules/pry

/lib/puppet/parser/functions/pry.rb @ line 4

#<Module:0xfb588d1>#real_function_pry:

2: newfunction(:pry) do |args|

3: require 'pry'

=> 4: binding.pry

5: end

[1] pry(#<Puppet::Parser::Scope>)>

puppet agent -t puppetserver foreground

…

Puppet Server has successfully started and is now ready

to handle requests

exit… Puppet Compiled Catalog for xxx.example.com in y.z seconds

Info: Caching catalog for xxx.example.com

Info: Applying configuration version 'XXX'


where to go for help● IRC #puppet / #puppet-dev

● slack puppetcommunity.slack.com#pug#puppet

● google group / mail listhttps://groups.google.com/forum/#!forum/puppet-users

● PUGhttps://www.meetup.com/Seattle-Puppet-Meetup/

70


Summary

Puppet is an HTTPS service

End-to-end (gethostbyname, nc mtr)

OpenSSL is your friend (x509,crl, verify, s_client

make a debug class

remember scope

basic UNIX permissions

71

puppetconf 2016: puppet troubleshooting – thomas uphill, wells fargo

Technology