DTrace integration and

quick startruBSD

Dec 2013

Veniamin Gvozdikov vg@FreeBSD.org

What is DTrace?

• Dynamic tracing framework

• Works on Solaris/FreeBSD/OSX/QNX/Linux

!2

Who uses it?

• Apple’s Xcode Instruments

• Cloud Analytics in Joyent SmartDataCenter

• Oracle’s ZFS Storage Appliance analytics

• Brendan’s Flame Graphs

• Command line DTraceToolkit

!3

Xcode Instruments

!4

Cloud Analytics

!5

Oracle’s ZFS-SA analytics

!6

Flame Graphs

!7

DTraceToolkit

!8

Terminology

• probe

• provider

• module

!9

How it works?dtrace (1)

kernel space

user spacelibdtrace

DTrace

!10

/dev/dtrace

libproc

Providers

dtrace: kernel space

fbt:

io:

syscall:

tcp:

dtmalloc:

lockstat:

profile:

!11

and others:

dtrace(1)• Command line frontend

• Shows available probes/providers

• Enable probes and attach to processes

• D Language compiler

• Works via libdtrace and libproc

• ETC…!12

Available probes

!13

To see all probes on the system you need to run: # dtrace -l [ … many lines … ]

To see probes by dtrace provider: # dtrace -l -n 'dtrace:::' ID PROVIDER MODULE FUNCTION NAME 1 dtrace BEGIN 2 dtrace END 3 dtrace ERROR

D program structure

provider:module:function:probe / predicates / { action; }

D language is like C and AWK:

!14

First D program

!15

# dtrace -q -n ' > syscall::open:entry { > printf("%s\t%s\n", execname, copyinstr(arg0)); > } > profile:::tick-1s { > exit(0); > } > ' !tmux /dev/null sh /etc/libmap.conf sh /var/run/ld-elf.so.hints sh /lib/libedit.so.7 sh /lib/libncurses.so.8 sh /lib/libc.so.7

Built-in Variables (not all)

args pid probeprov

errno probefunc timestamp

execname probemod vtimestamp

!16

Aggregating FunctionsFunction Name Arguments

count none

sum scalar expression

avg scalar expression

min scalar expression

max scalar expression

lquantize scalar expression, lower bound, upper bound, step value

quantize scalar expression

!17

Tracing applications

!18

• Trace with pid provider

• Trace with USDT (Userland Statically Defined Traces)

DTrace integration examples

Please clone repo (seeing code is required for next slides):

http://clck.ru/8wWyv

!19

int length_fast(char *); int length_slow(char *); int main(void) { int counter = 0; int c_loop = 0; char *Lines[] = { "x", "tt", "Test", NULL }; char *line; printf("%10s %4s\n", "STRING", "LEN"); while (15 > c_loop) { while (NULL != Lines[counter]) { line = Lines[counter]; printf("%10s: %02d\n", line, length_fast(line)); printf("%10s: %02d\n", line, length_slow(line)); ++counter; } ++c_loop; counter = 0; sleep(c_loop); } ! return 0; }

!20

#00 pid provider

!21

pid$target::length_*:entry { self->ts = timestamp; printf("String: %s\n", copyinstr(arg0)); }

00_script.sh

pid$target::length_*:return /self->ts/ { @num[probefunc] = count(); @sum[probefunc] = sum(timestamp - self->ts); @q[probefunc] = quantize(timestamp - self->ts); ! self->ts = 0; } !profile:::tick-10sec { exit(0); }

!22

END { printf("Count of call functions:\n"); printa(@num); printf("Time of call functions:\n"); printa(@sum); printf("Distribution of times by functions:\n"); printa(@q); }

!23

Count of call functions: ! length_fast 6 length_slow 6 !Time of call functions: ! length_fast 434394 length_slow 500954

Distribution of times by functions: ! length_fast value ------------- Distribution ------------- count 2048 | 0 4096 |@@@@@@@@@@@@@@@@@@@@ 3 8192 |@@@@@@@ 1 16384 | 0 32768 | 0 65536 | 0 131072 |@@@@@@@@@@@@@ 2 262144 | 0 ! length_slow value ------------- Distribution ------------- count 4096 | 0 8192 |@@@@@@@ 1 16384 |@@@@@@@ 1 32768 |@@@@@@@@@@@@@ 2 65536 | 0 131072 |@@@@@@@@@@@@@ 2 262144 | 0

result

!24

#01 First USDT providerdtrace_demo.d:

provider demo { probe start(); probe stop(); };

!25

Generate header file: # dtrace -h -s dtrace_demo.d

#01 Adding probes while (1) { if (DEMO_START_ENABLED()) DEMO_START(); while (NULL != Lines[counter]) { [ … lines … ] } counter = 0; sleep(1); if (DEMO_STOP_ENABLED()) DEMO_STOP(); }

!26

Compile applications

!27

Make object file: # clang -c main.c -DENABLE_DTRACE !Run dtrace(1) on object file: # dtrace -G -s dtrace_demo.d main.o !Link objects to application: # clang -lelf main.o dtrace_demo.o -o demo

#01 Trace application#!/usr/sbin/dtrace -Zs !#pragma D option quiet !demo*:::start { self->ts = timestamp; } !demo*:::stop { printf("time: %d\n", timestamp - self->ts); self->ts = 0; }

!28

01_script.sh# scripts/01_script.sh time: 1006057576 time: 1008003117 time: 1007984334 time: 1008002385 time: 1034989225 time: 1007995235 time: 1008000837 time: 1007991571 time: 1007992224 time: 1007998339

result

#02 Probes with arguments

provider demo { probe start(char *); probe stop(int); };

dtrace_demo.d:

!29

Generate header file: # dtrace -h -s dtrace_demo.d

#02 Adding probesint length_fast(char *string) { int i; ! if (DEMO_START_ENABLED()) DEMO_START(string); ! for (i = 0; '\0' != *string; ++i) { ++string; } if (DEMO_STOP_ENABLED()) DEMO_STOP(i); return i; }

!30

int length_slow(char *string) { int i; if (DEMO_START_ENABLED()) DEMO_START(string); ! for (i = 0; '\0' != *string; ++i) { ++string; usleep(i*i); } if (DEMO_STOP_ENABLED()) DEMO_STOP(i); return i; }

#!/usr/sbin/dtrace -Zs !#pragma D option quiet !BEGIN { self->start = timestamp; self->ts = 0; printf("Start tracing....\n\n"); } !demo*:::start { self->ts = timestamp; printf("String: %10s ", copyinstr(arg0)); }

!31

!demo*:::stop /self->ts/ { printf("len: %2d time: %6d \n", arg0, timestamp - self->ts); @[arg0] = quantize((timestamp - self->ts) / 1000); self->ts = 0; } !END { printf("Time %d.%d sec\n", (timestamp - self->start)/1000/1000/1000, (timestamp - self->start)/1000/1000/100); }

!32

# scripts/02_script.sh Start tracing.... !String: tt len: 2 time: 24425 String: Test len: 4 time: 5131 String: Test len: 4 time: 39085 String: x len: 1 time: 12314 String: x len: 1 time: 11881 String: tt len: 2 time: 4949 String: x len: 1 time: 11692 String: x len: 1 time: 8848 String: tt len: 2 time: 4430 String: tt len: 2 time: 24967 String: Test len: 4 time: 4561 String: Test len: 4 time: 37770 ^C Time 12.126 sec ![ … skipped distributions …]

result

#03 Evolution interface

!33

typedef struct { int dummy; } dline_t; typedef struct { int dummy; } demo_t; !provider demo { probe start(); probe stop(dline_t *dp) : (demo_t *dp); }; !#pragma D attributes Evolving/Evolving/ISA provider demo provider #pragma D attributes Private/Private/Unknown provider demo module #pragma D attributes Private/Private/Unknown provider demo function #pragma D attributes Private/Private/ISA provider demo name #pragma D attributes Evolving/Evolving/ISA provider demo args

dtrace_demo.d:

#03 Fill structureint length_fast(char *string) { dline_t data; int i; if (DEMO_START_ENABLED()) { data.mystr = string; DEMO_START(); } [ … some action … ] if (DEMO_STOP_ENABLED()) { data.len = i; DEMO_STOP(&data); } return i; }

typedef struct { const char *mystr; int len; } dline_t;

demo_provider.h:

!34

#03 First DTrace librarytypedef struct { char *mystr; int len; } dline_t; !typedef struct { string str; int len; } demo_t; !#pragma D binding "1.7" translator translator demo_t < dline_t *p > { str = copyinstr(*(uintptr_t *)copyin((uintptr_t)&p->mystr, sizeof (char*))); len = *(uint64_t *)copyin((uintptr_t)&p->len, sizeof (p->len)); };

!35

demo.d:

copy demo.d to /usr/lib/dtrace!or run:!# dtrace -L /path/to/dtrace/libs some args

#03 Access to structure

demo*:::stop /self->ts/ { printf("String: %10s len: %2d time: %6d\n", args[0]->str, args[0]->len, timestamp - self->ts); @[args[0]->str] = quantize((timestamp - self->ts) / 1000); self->ts = 0; }

03_script.sh:

!36

# scripts/03_script.sh Start tracing.... !String: x len: 1 time: 5178 String: x len: 1 time: 3485 String: tt len: 2 time: 1415 String: tt len: 2 time: 334482 String: Test len: 4 time: 1673 String: Test len: 4 time: 899032 ^C Time 2.20 sec [ … skipped distributions … ]

!37

result

libusdt

• Creates DTrace probes at runtime

• Bindings to Lua/Ruby/node.js/Perl

!38

libusdt and tarantool

Please clone repo recursively from dtrace branch:

http://clck.ru/8wX24See README.DTrace for more details

!39

lua usdt

provider = usdt.provider("init_script", "tarantool_box") probe = provider:probe("dt_hello", "entry", "char *", "int") provider:enable() !function dt_hello(mystr) probe:fire(mystr, string.len(mystr)) end

init.lua:

!40

# dtrace -l -n 'init_script*:::' ID PROVIDER MODULE FUNCTION NAME 249651 init_script18132 tarantool_box dt_hello entry

!41

# dtrace -q -n 'init_script*:::entry { printf("String %10s len: %02d\n", copyinstr(arg0), arg1) } tick-2s {exit(0);}' String Test len: 04 String tt len: 02 String xxx len: 03

localhost> for _, val in ipairs({"Test", "tt", "xxx"}) do dt_hello(val) end --- ...

Example

References• http://dtrace.org/

• http://wiki.FreeBSD.org/

• http://www.dtracebook.com

• http://docs.oracle.com/cd/E18752_01/html/819-5488/docinfo.html

• http://www.youtube.com/watch?v=i47XhPsAVTQ

• http://dev.mysql.com/doc/refman/5.6/en/dba-dtrace-server.html

• http://www.joyent.com/developers/resources/cloud-analytics-in-joyent-smartdatacenter

• http://www.brendangregg.com/dtracetoolkit.html

• https://github.com/brendangregg/dtrace-cloud-tools

• https://github.com/brendangregg/FlameGraph

• https://github.com/davepacheco/mod_usdt

• https://github.com/chrisa/libusdt

• https://github.com/chrisa/lua-usdt