sensu + graphite を1年運⽤してみて #sensucasual

23
Confiden’al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora’on. All Rights Reserved. Sensu Casual Talks #1 ヤフー株式会社 渡邉 貴志 [email protected]

Upload: yahoo

Post on 28-Nov-2014

2.697 views

Category:

Internet


2 download

DESCRIPTION

9/17 の Sensu Casual Talks #1 で発表された資料です。 http://www.zusaar.com/event/7607003

TRANSCRIPT

  • 1. Confiden'al :Discussion purpose only Sensu Casual Talks #1 [email protected] Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved.
  • 2. P2 Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. OS & Linux, FreeBSD PCM, ReRAM, MRAM, ...)
  • 3. Sensu + Graphite 1 - - Confiden'al :Discussion purpose only P3 Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved.
  • 4. P4 2011 IaaS HV Nagios Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved.
  • 5. P5 2013OpenStack HV 5000 Sensu + Graphite Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved.
  • 6. Sensu + Graphite P6 Nagios API Chef cookbook Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. portertech/chef-monitor sensu/sensu-chef subscriber role LWRP
  • 7. Confiden'al :Discussion purpose only P7 Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. 1
  • 8. 1P8 Hypervisor x N sensu-client redissensu-server Confiden'al :Discussion purpose only graphite-web conetxt-hashing replication=1 Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. All-in-one Sensu Server x 1 rabbitmq-server sensu-apisensu-dashboard carbon-relay server x 1 carbon-relay carbon-cache server x 3 carbon-cache
  • 9. 1P9 Hypervisor x N sensu-client Xeon L5640 2.13GHz x 1 DDR3-1333 2GB x 3 300GB 15k SAS HDD x 4 (RAID1+0) Intel 5520 + IOH-36D redissensu-server Confiden'al :Discussion purpose only Sensu graphite-web conetxt-hashing replication=1 Xeon E5-2630L 2.00GHz x 2 DDR3-1600 8GB x 8 300GB 10k SAS HDD x 4 (RAID1+0) Intel C602 Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. All-in-one Sensu Server x 1 rabbitmq-server sensu-apisensu-dashboard carbon-relay server x 1 carbon-relaycarbon-cache server x 3 carbon-cache
  • 10. 1P10 200 500 Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved.
  • 11. 1P11 Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. sensu LOG LEVEL = warn max open files server Checks10 per 60sec / RAID etc... 11 graphite
  • 12. 1P12 Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. graphite relay consistent hash 1 cache HDD IO tmpfs 55GB cron aggregator
  • 13. Confiden'al :Discussion purpose only P13 Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. 2
  • 14. 2P14 redissensu-server Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. All-in-one Sensu Server x 3 rabbitmq-server sensu-apisensu-dashboard carbon-relay server x 2 carbon-relay carbon-cache server x 3 carbon-cache Hypervisor x N sensu-client graphite-web conetxt-hashing replication=1
  • 15. 2P15 Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. Sensu All-in-one 3 redis 1 rabbitmq Graphite carbon-relay 2
  • 16. 2P16 Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. 1000 ... ntp sensu-server
  • 17. Confiden'al :Discussion purpose only P17 Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. 3
  • 18. 3 redis server x 1 Confiden'al :Discussion purpose only P18 carbon-cache x N Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. dashboard server x 1 sensu server x 30 rabbitmq server x 3 rabbitmq-server sensu-apiuchiwa sensu-server redis carbon-relay server x 2 carbon-relay x N carbon-cache server x 6 Hypervisor x N sensu-clienthaproxygraphite-web carbon-relay x N gluster conetxt-hashing replication=1 conetxt-hashing replication=1 replication=2
  • 19. 3P19 3000+ VM 1HV 10VM+ VM API Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved.
  • 20. 3P20 Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. sensu sensu-server on VM redis ... UI uchiwa v0.2+
  • 21. 3P21 Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. graphite LB haproxy relay daemon carbon-relay = CPU bound carbon-cache = IO bound megacarbon v0.10+ LOG_WRITES=false / LOG_CACHE_HITS = false max open files whisper ceres (v0.10+) whisper cron
  • 22. Confiden'al :Discussion purpose only P22 Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved.
  • 23. P23 sensu + graphite 1 Confiden'al :Discussion purpose only Copyright (C) 2012 Yahoo Japan Corpora'on. All Rights Reserved. CM carbon-cache