Topic 200: Capacity Planning

Object 1: Measure and Troubleshoot Resource Usage

4

System resources

CPU

Memory I/O

5

Linux performance metrics● Processor metrics

● Memory metrics

● Block device metrics

● Network interface metrics

6

Monitoring toolsبسته های مورد نیاز●

● sysstat● iostat, sar, mpstat, ...

● procps● free, uptime, vmstat, w, ...

10

Process management ادامه ● Context switching

● Interrupt handling

● Process state (runing,rtopped,intruptible,unintruptible,zombie)

fork()TASK_STOPPED

TASK_UNINTERRUPTIBLE

TASK_INTERRUPTIBLE

TASK_RUNNING(ready)

TASK_ZOMBIE

exit()

Processor

TASK_RUNNING

11

Processor metrics● CPU utilization (Exceed 80%=bottleneck)

● User time, System time,Waiting, Idle time, Nice time

● Load avarage

● Runable processes (Excedd 10 time the amount of prcessors)

● Blocked (Uninterruptible: like process waiting for I/O)

● Context switch, Interrupts(software bottleneck)

13

vmstat$ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 262008 544552 21036 392400 1 4 173 142 136 332 15 5 77 2

$ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 262008 544552 21036 392400 1 4 173 142 136 332 15 5 77 2

Process (procs) r: The number of processes waiting for runtime b: The number of processes in uninterruptable sleepMemory swpd: The amount of virtual memory used (KB)

free: The amount of idle memory (KB) buff: The amount of memory used as buffers (KB)

cache: The amount of memory used as cache (KB)Swap si: Amount of memory swapped from the disk (Kbps)

so: Amount of memory swapped to the disk (Kbps)IO bi: Blocks sent to a block device (blocks/s)

bo: Blocks received from a block device (blocks/s)System in: The number of interrupts per second, including the clock cs: The number of context switches per second

Process (procs) r: The number of processes waiting for runtime b: The number of processes in uninterruptable sleepMemory swpd: The amount of virtual memory used (KB)

free: The amount of idle memory (KB) buff: The amount of memory used as buffers (KB)

cache: The amount of memory used as cache (KB)Swap si: Amount of memory swapped from the disk (Kbps)

so: Amount of memory swapped to the disk (Kbps)IO bi: Blocks sent to a block device (blocks/s)

bo: Blocks received from a block device (blocks/s)System in: The number of interrupts per second, including the clock cs: The number of context switches per second

14

sar$ sar -q 02:55:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked03:05:01 PM 0 566 1.83 2.01 2.16 0

$ sar -q 02:55:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked03:05:01 PM 0 566 1.83 2.01 2.16 0

$ sar -P ALLAverage: CPU %user %nice %system %iowait %steal %idleAverage: all 15.21 0.03 5.20 1.74 0.00 77.81Average: 0 16.22 0.03 5.25 1.95 0.00 76.54Average: 1 16.14 0.03 5.38 1.97 0.00 76.48Average: 2 14.24 0.04 5.02 1.52 0.00 79.18Average: 3 14.24 0.03 5.16 1.51 0.00 79.06

$ sar -P ALLAverage: CPU %user %nice %system %iowait %steal %idleAverage: all 15.21 0.03 5.20 1.74 0.00 77.81Average: 0 16.22 0.03 5.25 1.95 0.00 76.54Average: 1 16.14 0.03 5.38 1.97 0.00 76.48Average: 2 14.24 0.04 5.02 1.52 0.00 79.18Average: 3 14.24 0.03 5.16 1.51 0.00 79.06

15

sar$ sar -wLinux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)

01:51:22 AM LINUX RESTART

01:55:01 AM proc/s cswch/s02:05:01 AM 1.14 7087.0902:15:01 AM 1.19 6955.2302:25:01 AM 1.12 6855.2102:35:01 AM 1.21 6311.1802:45:01 AM 1.20 6006.53

$ sar -wLinux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)

01:51:22 AM LINUX RESTART

01:55:01 AM proc/s cswch/s02:05:01 AM 1.14 7087.0902:15:01 AM 1.19 6955.2302:25:01 AM 1.12 6855.2102:35:01 AM 1.21 6311.1802:45:01 AM 1.20 6006.53

16

Other command$ mpstat -P ALLLinux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)

04:50:28 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle04:50:28 PM all 14.94 0.03 4.62 1.80 0.01 0.47 0.00 0.00 78.13

$ mpstat -P ALLLinux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)

04:50:28 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle04:50:28 PM all 14.94 0.03 4.62 1.80 0.01 0.47 0.00 0.00 78.13

$ iostat -xLinux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle 14.92 0.03 5.10 1.79 0.00 78.15

$ iostat -xLinux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle 14.92 0.03 5.10 1.79 0.00 78.15

17

Linux memory architecture● Physical and virtual memory

● Virtual memory manager ● Page frame● Page frame reclaiming (Swap ,...)● Buddy system

18

Memory metrics● Free memory

● Swap usage

● Buffer and caches

● Slabs (kernel usage of memory)

● Active versus inactive memory ● Inactive memory is a likely candidate to be swapped out to disk

19

free, top, vmstat$ top -b -n 1 | head -5top - 16:54:00 up 5:39, 3 users, load average: 0.93, 0.65, 0.58Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie%Cpu(s): 14.9 us, 4.6 sy, 0.0 ni, 78.2 id, 1.8 wa, 0.0 hi, 0.5 si, 0.0 stKiB Mem: 2051244 total, 1600512 used, 450732 free, 29236 buffersKiB Swap: 4954776 total, 254804 used, 4699972 free, 445900 cached

$ top -b -n 1 | head -5top - 16:54:00 up 5:39, 3 users, load average: 0.93, 0.65, 0.58Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie%Cpu(s): 14.9 us, 4.6 sy, 0.0 ni, 78.2 id, 1.8 wa, 0.0 hi, 0.5 si, 0.0 stKiB Mem: 2051244 total, 1600512 used, 450732 free, 29236 buffersKiB Swap: 4954776 total, 254804 used, 4699972 free, 445900 cached

$ vmstat -aprocs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free inact active si so bi bo in cs us sy id wa 0 1 254804 453648 613696 844384 1 3 155 129 309 480 15 5 78 2

$ vmstat -aprocs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free inact active si so bi bo in cs us sy id wa 0 1 254804 453648 613696 844384 1 3 155 129 309 480 15 5 78 2

20

sar$ sar -rLinux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)

01:51:22 AM LINUX RESTART

01:55:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact02:05:01 AM 289476 1761768 85.89 75248 547476 3017772 43.07 1141260 480772

$ sar -rLinux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU)

01:51:22 AM LINUX RESTART

01:55:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact02:05:01 AM 289476 1761768 85.89 75248 547476 3017772 43.07 1141260 480772

25

Block device metrics● Iowait

● Average queue length

● Average wait

● Transfers per second

● Blocks read/write per second

● Kilobytes per second read/write

26

iostat

$ iostat -p ALL 1 1$ iostat -p ALL 1 1

$ iostat -x 1 1Linux 3.2.0-4-amd64 (laptop) 05/29/2014 _x86_64_ (4 CPU)avg-cpu: %user %nice %system %iowait %steal %idle 8.62 0.03 2.61 2.15 0.00 86.60Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %utilsda 1.44 5.36 3.35 4.37 196.43 59.67 66.38 0.25 32.64 12.89 47.76 6.00 4.63

$ iostat -x 1 1Linux 3.2.0-4-amd64 (laptop) 05/29/2014 _x86_64_ (4 CPU)avg-cpu: %user %nice %system %iowait %steal %idle 8.62 0.03 2.61 2.15 0.00 86.60Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %utilsda 1.44 5.36 3.35 4.37 196.43 59.67 66.38 0.25 32.64 12.89 47.76 6.00 4.63

27

sar

$ sar -bLinux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU)

12:34:50 PM LINUX RESTART

12:35:04 PM tps rtps wtps bread/s bwrtn/s12:45:01 PM 29.74 21.93 7.81 1389.04 292.9012:55:01 PM 5.90 1.34 4.56 32.94 97.9001:05:01 PM 20.23 13.89 6.35 760.67 698.3001:15:01 PM 83.81 72.50 11.31 1943.13 1598.7301:25:01 PM 5.55 1.47 4.08 39.11 100.80

$ sar -bLinux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU)

12:34:50 PM LINUX RESTART

12:35:04 PM tps rtps wtps bread/s bwrtn/s12:45:01 PM 29.74 21.93 7.81 1389.04 292.9012:55:01 PM 5.90 1.34 4.56 32.94 97.9001:05:01 PM 20.23 13.89 6.35 760.67 698.3001:15:01 PM 83.81 72.50 11.31 1943.13 1598.7301:25:01 PM 5.55 1.47 4.08 39.11 100.80

28

sar

$ sar -dLinux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU)

12:34:50 PM LINUX RESTART

12:35:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util12:45:01 PM dev8-0 29.74 1389.04 292.90 56.55 0.96 33.07 5.69 16.9312:55:01 PM dev8-0 5.90 32.94 97.90 22.17 0.07 11.91 5.40 3.1901:05:01 PM dev8-0 20.23 760.67 698.30 72.10 0.43 21.23 3.50 7.09

$ sar -dLinux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU)

12:34:50 PM LINUX RESTART

12:35:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util12:45:01 PM dev8-0 29.74 1389.04 292.90 56.55 0.96 33.07 5.69 16.9312:55:01 PM dev8-0 5.90 32.94 97.90 22.17 0.07 11.91 5.40 3.1901:05:01 PM dev8-0 20.23 760.67 698.30 72.10 0.43 21.23 3.50 7.09

$ ls -l /dev/sdabrw-rw---T 1 root disk 8, 0 Apr 23 12:34 /dev/sda$ ls -l /dev/sdabrw-rw---T 1 root disk 8, 0 Apr 23 12:34 /dev/sda

31

Network interface metrics● Packets received and sent, Bytes received and sent

● Collisions per second

● Packets dropped

● Overruns (ran out of buffer space)

● Errors

32

ifconfig

$ ifconfig eth0wlan0 Link encap:Ethernet HWaddr 00:08:ca:45:d0:76 inet addr:192.168.13.51 Bcast:192.168.13.255 Mask:255.255.255.0 inet6 addr: fe80::208:caff:fe45:d076/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:272597 errors:0 dropped:0 overruns:0 frame:0 TX packets:259457 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:350358017 (334.1 MiB) TX bytes:48277958 (46.0 MiB)

$ ifconfig eth0wlan0 Link encap:Ethernet HWaddr 00:08:ca:45:d0:76 inet addr:192.168.13.51 Bcast:192.168.13.255 Mask:255.255.255.0 inet6 addr: fe80::208:caff:fe45:d076/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:272597 errors:0 dropped:0 overruns:0 frame:0 TX packets:259457 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:350358017 (334.1 MiB) TX bytes:48277958 (46.0 MiB)

33

sar$ sar -n DEVLinux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU)

12:34:50 PM LINUX RESTART

12:35:04 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s12:45:01 PM lo 0.07 0.07 0.01 0.01 0.00 0.00 0.0012:45:01 PM wlan0 12.83 12.76 13.38 2.39 0.00 0.00 0.0012:45:01 PM kvnet 7.97 8.28 8.10 1.10 0.00 0.00 0.0012:45:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

$ sar -n DEVLinux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU)

12:34:50 PM LINUX RESTART

12:35:04 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s12:45:01 PM lo 0.07 0.07 0.01 0.01 0.00 0.00 0.0012:45:01 PM wlan0 12.83 12.76 13.38 2.39 0.00 0.00 0.0012:45:01 PM kvnet 7.97 8.28 8.10 1.10 0.00 0.00 0.0012:45:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

34

netstat

# netstat -antupActive Internet connections (servers and established)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 192.168.13.51:80 192.168.13.52:53930 ESTABLISHED 27398/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53931 ESTABLISHED 27399/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53930 TIME_WAIT - tcp 0 0 192.168.13.60:80 192.168.13.52:53931 SYN_RECV 27559/apache2

# netstat -antupActive Internet connections (servers and established)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 192.168.13.51:80 192.168.13.52:53930 ESTABLISHED 27398/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53931 ESTABLISHED 27399/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53930 TIME_WAIT - tcp 0 0 192.168.13.60:80 192.168.13.52:53931 SYN_RECV 27559/apache2

# netstat -s # netstat -s

Object 2: Predict Future Resource Needs

36

Monitor resource usage

Monitoring Solution

Collectd

Munin

Nagios

Cacti

Mrtg

37

Collectdنصب بسته های مورد نیاز●

مسیر ها●

# apt-get install collectd librrds-perl libconfig-general-perl libhtml-parser-perl libregexp-common-perl libjson-perl # apt-get install collectd librrds-perl libconfig-general-perl libhtml-parser-perl libregexp-common-perl libjson-perl

Config file: /etc/collectd/{collectd.conf,collection.conf} Plugins : /usr/lib/collectd Data: /var/lib/collectd

Config file: /etc/collectd/{collectd.conf,collection.conf} Plugins : /usr/lib/collectd Data: /var/lib/collectd

38

Setting Options ● Global options (Interval, FQDNLookup, …)

● Plugin options (documented on man collectd.conf )“ ”

BaseDir "/var/lib/collectd" PIDFile "/run/collectd.pid" Interval 10.0 (if changed delete all rrd files)

LoadPlugin cpu LoadPlugin df

<LoadPlugin df> Interval 3600 </LoadPlugin>

BaseDir "/var/lib/collectd" PIDFile "/run/collectd.pid" Interval 10.0 (if changed delete all rrd files)

LoadPlugin cpu LoadPlugin df

<LoadPlugin df> Interval 3600 </LoadPlugin>

39

Plugins ● Plugins

● Input (read) Plugins (CPU, Memory)● Output (write) Plugins (RRDtool,CSV)● Read and Write Plugins (Network)● Log Plugins (LogFile,Syslog)# vi /etc/collectd/collectd.conf...LoadPlugin apacheLoadPlugin cpuLoadPlugin dfLoadPlugin networkLoadPlugin interface

# vi /etc/collectd/collectd.conf...LoadPlugin apacheLoadPlugin cpuLoadPlugin dfLoadPlugin networkLoadPlugin interface

41

Creating Graphs● Debian

● Collection3

# apt-get install apache2-mpm-worker (Wheezy) # apt-get install apache2-mpm-worker (Wheezy)

# cp -r /usr/share/doc/collectd/examples/collection3 /var/www/ # cp -r /usr/share/doc/collectd/examples/collection3 /var/www/

# apt-get install apache2 (Jessie) # apt-get install apache2 (Jessie)

# a2enmod cgi # vi /etc/apache2/sites-enabled/default <Directory /var/www/collection3> Options +ExecCgi AddHandler cgi-script .cgi .pl </Directory>

# a2enmod cgi # vi /etc/apache2/sites-enabled/default <Directory /var/www/collection3> Options +ExecCgi AddHandler cgi-script .cgi .pl </Directory>