let's begin resource monitoring with munin 2011 1209 zem_distribution

96
Munin Munin 今日から始める 今日から始める リソ ス監視 リソス監視 Munin User Group Japan http://munin.jp/ @zembutsu Dec 9, 2011 at IBM@SHIBUYA MARKCITY

Upload: masahito-zembutsu

Post on 07-Nov-2014

2.571 views

Category:

Technology


1 download

DESCRIPTION

Munin で始めるリソースモニタリング。Munin User Group Japan の紹介、自作プラグイン(迷惑メール数カウントフィルタ)のサンプル、各種リソースモニタのサンプル。おまけでMuninのインストール方法。

TRANSCRIPT

Page 1: Let's begin resource monitoring with munin 2011 1209 zem_distribution

Munin でMunin で、今日から始める今日から始めるリソ ス監視リソース監視Munin User Group Japanp phttp://munin.jp/@zembutsu@Dec 9, 2011 at IBM@SHIBUYA MARKCITY

Page 2: Let's begin resource monitoring with munin 2011 1209 zem_distribution

About meAbout me• @zembutsu 前佛雅人

– 活動:クラウド界隈に出没クラウドは人生(キリッ

活動:クラウド界隈に出没• Open Cloud Campus, JEUG(Eucalyptus ユーザ会), JAWS-UG等• Perl 使い、日本酒大好き• 本業はat+link専用サーバサービスの運用サポート等等• 本業はat+link専用サーバサービスの運用サポート等等

– 最近の興味• データマイニング、統計・解析、MongoDB• Last Exile Fam、北米版の日本アニメ鑑賞

– http://www.slideshare.net/zembutsu• G-CLOUDMagazine G-CLOUDMagazine 2011 SummerG CLOUD Magazine G CLOUD Magazine 2011 Summer

2010年8月6日発売 2011年6月29日発売1章~3章AWS入門記事 5章RightScale

Page 3: Let's begin resource monitoring with munin 2011 1209 zem_distribution

http://munin jp/http://munin.jp/Muninユーザ会が出来ました。サイトもオ プンです ( ∀ )ノサイトもオープンですヽ(・∀・)ノ

Page 4: Let's begin resource monitoring with munin 2011 1209 zem_distribution

http://munin jp/http://munin.jp/

Page 5: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 6: Let's begin resource monitoring with munin 2011 1209 zem_distribution

とある会社の迷惑メール到達状況をMuninプラグインで視覚化したもの

Page 7: Let's begin resource monitoring with munin 2011 1209 zem_distribution

#!/bin/sh### Plugin to monitor SAPM email detection and reject counter# 2011/11/28

M i プラグインは簡単#%# family=auto#%# capabilities=autoconf

Muninプラグインは簡単迷惑メールカウントは50行のシェルスクリプト

if [ "$1" = "autoconf" ] ; thenif [ -n "$SPAMS" ] ; then

echo "yes"

50行のシ ルスクリプト

yelse

echo "¥$SPAMS not defined."fiexit

fi

if [ "$1" " fi " ] hif [ "$1" = "config" ] ; thenecho "graph_args --base 1000 -r --lower-limit 0";#echo "graph_args --base 1000 -r --lower-limit 0 --units-exponent 3";

h " h titl SPAM E il R j t C t b t "echo "graph_title SPAM E-mails Reject Counter, by type";echo "graph_category sendmail";echo "graph_info This graph shows procmail filter counter.";echo 'graph printf %3 0lf'echo graph_printf %3.0lfecho 'graph_vlabel SPAMs'echo "graph_order $SPAM"

Page 8: Let's begin resource monitoring with munin 2011 1209 zem_distribution

FIRSTSPAM=1;for SPAM in $SPAMS; do

echo "${SPAM}.label $SPAM"echo "${SPAM}.info procmail recipe by $SPAM"echo "${SPAM}.type GAUGE"if [ $FIRSTSPAM -eq 1 ] ; then

h "${SPAM} d AREA" グラフの定義グラフの定義echo "${SPAM}.draw AREA"export FIRSTSPAM=0;

elseecho "${SPAM} dra STACK"

グラフの定義(rrdtool)

echo "${SPAM}.draw STACK"fi

done ;

exitfi

for SPAM in $SPAMS ; do {grep $SPAM /var/lib/munin/plugin-state/SpamFiltered.state

}}

done;値を取得

( Muninが fetch )

Page 9: Let's begin resource monitoring with munin 2011 1209 zem_distribution

4,9,14,19,24,29,34,39,44,49,54,59 * * * * /usr/share/munin/plugins/countSpamFiltered.sh > /var/lib/munin/plugin-t t /S Filt d t tstate/SpamFiltered.state

Cronで情報取得Cronで情報取得(muninユーザ権限ではアクセスできないファイルのため 中間ファイルを作成)中身#!/bin/sh

# by zembutsu@link co jp 2011/11/28

め、中間ファイルを作成)中身

# by [email protected] 2011/11/28# env spam.body spam.from spam.received

for SPAM in body from received x info Subject Body null body Nov class mailerfor SPAM in body from received x-info Subject_Body_null body-Nov class mailer messageid reply return body-DotAsia RegacyAddress Subject-Word X-Mailer-Outlook-Forged body-Decddo

echo -n "${SPAM}.value "grep ^Message- /var/spool/procmail/spam.${SPAM}.log | wc -l

done

Page 10: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 11: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 12: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 13: Let's begin resource monitoring with munin 2011 1209 zem_distribution

http://munin-monitoring.org/

Page 14: Let's begin resource monitoring with munin 2011 1209 zem_distribution

( ムーニン )( ム ニン )

Page 15: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 16: Let's begin resource monitoring with munin 2011 1209 zem_distribution

h // l d iki di / iki di / /3/3 /Odi h f jhttp://upload.wikimedia.org/wikipedia/commons/3/3c/Odin_hrafnar.jpg

Page 17: Let's begin resource monitoring with munin 2011 1209 zem_distribution

h // l d iki di / iki di / /3/3 /Odi h f jhttp://upload.wikimedia.org/wikipedia/commons/3/3c/Odin_hrafnar.jpg

Page 18: Let's begin resource monitoring with munin 2011 1209 zem_distribution

http://munin-monitoring.org/

Page 19: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 20: Let's begin resource monitoring with munin 2011 1209 zem_distribution

横から見た横から見たワタリガラス

よく見るとiMunin

Page 21: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 22: Let's begin resource monitoring with munin 2011 1209 zem_distribution

よくあるトラフ クの視覚化よくあるトラフィックの視覚化

Page 23: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 24: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 25: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 26: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 27: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 28: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 29: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 30: Let's begin resource monitoring with munin 2011 1209 zem_distribution

サンプルサンプルhttp://node1.pocketstudio.net/munin/

Page 31: Let's begin resource monitoring with munin 2011 1209 zem_distribution

複数サ バのリソ ス状況を複数サーバのリソース状況をWebブラウザ上に並列表示

Page 32: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 33: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 34: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 35: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 36: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 37: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 38: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 39: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 40: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 41: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 42: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 43: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 44: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 45: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 46: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 47: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 48: Let's begin resource monitoring with munin 2011 1209 zem_distribution

NodeNodeが生きてが生きていれば、

LoadAverageg8,000 も!!

Page 49: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 50: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 51: Let's begin resource monitoring with munin 2011 1209 zem_distribution

秒間アクセス数を表示

Page 52: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 53: Let's begin resource monitoring with munin 2011 1209 zem_distribution

Nginxの秒間アクセス数を

表示

Page 54: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 55: Let's begin resource monitoring with munin 2011 1209 zem_distribution

MySQLクエリMySQLクエリ毎に視覚化

Page 56: Let's begin resource monitoring with munin 2011 1209 zem_distribution

ピーク約46k ピ ク約query/sec

Page 57: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 58: Let's begin resource monitoring with munin 2011 1209 zem_distribution

メールの転送バイト数/sec

Page 59: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 60: Let's begin resource monitoring with munin 2011 1209 zem_distribution

HDD温度の推移

Page 61: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 62: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 63: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 64: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 65: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 66: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 67: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 68: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 69: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 70: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 71: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 72: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 73: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 74: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 75: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 76: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 77: Let's begin resource monitoring with munin 2011 1209 zem_distribution

総発電量総発電量

電力供給量

Page 78: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 79: Let's begin resource monitoring with munin 2011 1209 zem_distribution

電力使用率電力使用率

Page 80: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 81: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 82: Let's begin resource monitoring with munin 2011 1209 zem_distribution

たまにデ タたまにデータが跳ね上がる事が事が…

Page 83: Let's begin resource monitoring with munin 2011 1209 zem_distribution

長期トレンドでは変に欠損したり(;´Д`)

Page 84: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 85: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 86: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 87: Let's begin resource monitoring with munin 2011 1209 zem_distribution

h // i j /http://munin.jp/http://munin.jp/Munin User Group JapanMunin User Group Japan

Page 88: Let's begin resource monitoring with munin 2011 1209 zem_distribution

まとめまとめ

• Muninを使って、瞬時にボトルネ クを特定1. 瞬時にボトルネックを特定

2 継続的な状況の把握2. 継続的な状況の把握– 障害発生時だけじゃないお障害発生時だけじゃないお

運用管理の効率化に• 運用管理の効率化にもっと時間を有効に使おう!–もっと時間を有効に使おう!

88

Page 89: Let's begin resource monitoring with munin 2011 1209 zem_distribution

Thank you for listening!Thank you for listening!

•最後までおつきあいいただき、ありがとうございました!!ありがとうございました!!

• ContactsContacts– Munin User Group Japan

http://munin jp/• http://munin.jp/– 前佛雅人 (Masahito Zembutsu)

• Twitter: @zembutsu• Twitter: @zembutsu• E-mail: [email protected]

Page 90: Let's begin resource monitoring with munin 2011 1209 zem_distribution

Appendix How to install MuninAppendix - How to install Munin

Page 91: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 92: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 93: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 94: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 95: Let's begin resource monitoring with munin 2011 1209 zem_distribution
Page 96: Let's begin resource monitoring with munin 2011 1209 zem_distribution