Мониторинг. Опять, rootconf 2016
TRANSCRIPT
![Page 1: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/1.jpg)
МОНИТОРИНГ. ОПЯТЬ.Всеволод Поляков
![Page 3: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/3.jpg)
Что такое метрики?
![Page 4: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/4.jpg)
![Page 5: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/5.jpg)
![Page 6: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/6.jpg)
Успешность
![Page 7: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/7.jpg)
Количество
![Page 8: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/8.jpg)
Время
![Page 9: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/9.jpg)
![Page 10: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/10.jpg)
Взаимодействие
![Page 11: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/11.jpg)
Внутренние процессы
![Page 12: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/12.jpg)
Системные метрики
![Page 13: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/13.jpg)
Зачем нужны метрики?
![Page 14: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/14.jpg)
Алерты
![Page 15: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/15.jpg)
Аналитика
![Page 16: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/16.jpg)
![Page 17: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/17.jpg)
Graphite
![Page 18: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/18.jpg)
Default graphite architecture
![Page 19: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/19.jpg)
what?• RRD-like (gram.ly/gfsx)
• so.it.is.my.metric → /so/it/is/my/metric.wsp
• Fixed retention (by name\pattern)
• Fixed size (actually no)
![Page 20: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/20.jpg)
Retention and size• 1s:1d → 1 036 828 bytes
• 10s:10d → 1 036 828 bytes
• 1s:365d → 378 432 028 bytes (1 TB ~ 3 000)
• 10s:365d → 37 843 228 bytes (1 TB ~ 30 000)
whisper calc
![Page 21: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/21.jpg)
Retention and size• 10s:30d,1m:120d,10m:365d → 4 564 864 bytes
• 240 864 metrics in 1 TB
• aggregation: average, sum, min, max, and last.
• can be assign per metric
![Page 22: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/22.jpg)
How• terraform (https://www.terraform.io/)
• docker (https://www.docker.com/)
• ansible (https://www.ansible.com/)
• rocker (https://github.com/grammarly/rocker)
• rocker-compose (https://github.com/grammarly/rocker-compose)
![Page 23: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/23.jpg)
Default graphite architecture
![Page 24: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/24.jpg)
carbon-cache.py
• single-core
• many options in config file
• default
link
![Page 25: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/25.jpg)
architecturecarbon-cache.py
![Page 26: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/26.jpg)
Start load testing• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• MAX_CACHE_SIZE, MAX_UPDATES_PER_SECOND, MAX_CREATES_PER_MINUTE = inf
• defaults
• almost 1.5h to get limit :(
![Page 27: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/27.jpg)
carbon-cache.py cache size → 75k m\s
![Page 28: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/28.jpg)
![Page 29: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/29.jpg)
![Page 30: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/30.jpg)
results
• 75 000 m\s max
• 60 000 m\s flagman speed
• I\O :(
![Page 31: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/31.jpg)
Try to tune!
• WHISPER_SPARSE_CREATE = true (don’t allocate space on creation) non-linear I\O load.
• CACHE_WRITE_STRATEGY = sorted (default)
![Page 32: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/32.jpg)
cache size 1k → 195k m\s
![Page 33: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/33.jpg)
results
• 120 000 m\s flagman speed • cache flush problem :(
![Page 34: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/34.jpg)
Try to tune!
• CACHE_WRITE_STRATEGY = max will give a strong flush preference to frequently updated metrics and will also reduce random file-io.
![Page 35: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/35.jpg)
from 1k to 150k
![Page 36: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/36.jpg)
results
• 90 000 m\s flagman speed • cache flush problem :(
![Page 37: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/37.jpg)
Try to tune!
• CACHE_WRITE_STRATEGY = naive just flush. Better with random I\O.
![Page 38: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/38.jpg)
from 45k to 135k
![Page 39: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/39.jpg)
results
• 120 000 m\s flagman speed • still CPU
![Page 40: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/40.jpg)
sorted
max
naive
![Page 41: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/41.jpg)
• Maybe it’s I\O EBS limitation? → 512 GB disk.
• No.
![Page 42: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/42.jpg)
go-carbon
• multi-core single daemon
• written in golang
• not many options to tune :(
link
![Page 43: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/43.jpg)
Start load testing• m4.xlarge instance (4 CPU, 16 GB ram, 256 GB disk EBS gp2)
• retentions = 1s:1d
• max-size = 0
• max-updates-per-second = 0
• almost 1h to get limit :(
![Page 44: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/44.jpg)
1k → 130k m\s ~3k/min
![Page 45: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/45.jpg)
![Page 46: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/46.jpg)
results• 120 000 m\s flagman speed • but it’s without sparse. • try to implement
![Page 47: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/47.jpg)
try to tune! remaining := whisper.Size() - whisper.MetadataSize() whisper.file.Seek(int64(remaining-1), 0) whisper.file.Write([]byte{0}) chunkSize := 16384 zeros := make([]byte, chunkSize) for remaining > chunkSize { // if _, err = whisper.file.Write(zeros); err != nil { // return nil, err // } remaining -= chunkSize } if _, err = whisper.file.Write(zeros[:remaining]); err != nil { return nil, err }
![Page 48: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/48.jpg)
Уже есть в go-carbon
![Page 49: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/49.jpg)
180 000 m\s !
![Page 50: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/50.jpg)
![Page 51: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/51.jpg)
try to tune!
• max update operation = 1500
![Page 52: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/52.jpg)
results
• TLDR 210 000 - 240 000 m\s flagman speed
• 31 000 000 cache size!
![Page 53: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/53.jpg)
![Page 54: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/54.jpg)
try to tune!
• max update operation = 0
• input-buffer = 400 000
![Page 55: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/55.jpg)
results
• 270 000 m\s flagman speed
• 10-20kk cache size!
![Page 56: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/56.jpg)
![Page 57: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/57.jpg)
try to tune!
• vm.dirty_background_ratio=40
• vm.dirty_ratio=60
![Page 58: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/58.jpg)
300 000 req\s
![Page 59: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/59.jpg)
results
• 300 000 m\s flagman speed
• 180k+ m\s ±without cache
![Page 60: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/60.jpg)
Re:Lays
![Page 61: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/61.jpg)
Default graphite architecture
![Page 62: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/62.jpg)
arch forward
![Page 63: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/63.jpg)
arch named\regexp
![Page 64: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/64.jpg)
arch hash
![Page 65: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/65.jpg)
arch hash replicafactor: 2
![Page 66: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/66.jpg)
carbon-relay.py
• twisted based
• native
![Page 67: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/67.jpg)
Start load testing• c4.xlarge instance (4 CPU, 7.5 GB ram)
• ~1 Gb lan
• default parameters
• hashing
• 10 connections
![Page 68: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/68.jpg)
WTF!
![Page 69: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/69.jpg)
carbon-relay-ng• golang-based
• web-panel
• live-updates
• aggregators
• spooling
link
![Page 70: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/70.jpg)
<150 000 req\s
![Page 71: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/71.jpg)
carbon-c-relay
• написан на C
• advanced cluster management
![Page 72: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/72.jpg)
from 100 000 to 1 600 000 req\s
![Page 73: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/73.jpg)
1 400 000 flagman speed. Or not?
![Page 74: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/74.jpg)
Итак…go-carbon + carbon-c-relay = ♡
![Page 75: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/75.jpg)
![Page 76: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/76.jpg)
![Page 77: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/77.jpg)
![Page 78: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/78.jpg)
![Page 79: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/79.jpg)
![Page 80: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/80.jpg)
Контейнеры
![Page 81: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/81.jpg)
Всё перепутано
![Page 82: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/82.jpg)
Различия• Окружение
• Роль
• Трек (Модификатор)
• IP
• Датацентр
• Что-угодно
![Page 83: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/83.jpg)
Теги
![Page 84: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/84.jpg)
TSDB с тегами
• influxDB
• openTSDB (hbase)
• cyanite (cassandra)
• newTS (cassandra)
• Prometheus
![Page 85: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/85.jpg)
(cluster) influx, 130k metric\sувеличить график
![Page 86: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/86.jpg)
![Page 87: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/87.jpg)
openTSDB single instance + hbase cluster = upto 150k metric\s
![Page 88: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/88.jpg)
![Page 89: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/89.jpg)
![Page 90: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/90.jpg)
![Page 91: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/91.jpg)
![Page 92: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/92.jpg)
Compaction
![Page 93: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/93.jpg)
![Page 94: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/94.jpg)
Graphite
![Page 95: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/95.jpg)
![Page 96: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/96.jpg)
![Page 97: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/97.jpg)
![Page 98: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/98.jpg)
![Page 99: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/99.jpg)
Найти уникальное
![Page 100: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/100.jpg)
![Page 101: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/101.jpg)
![Page 102: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/102.jpg)
![Page 103: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/103.jpg)
![Page 104: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/104.jpg)
![Page 105: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/105.jpg)
![Page 106: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/106.jpg)
![Page 107: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/107.jpg)
Работает с Grafana
![Page 108: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/108.jpg)
Zipper
• https://github.com/grobian/carbonserver
• https://github.com/dgryski/carbonzipper
• https://github.com/dgryski/carbonapi
![Page 109: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/109.jpg)
![Page 110: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/110.jpg)
![Page 111: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/111.jpg)
ALSO
• https://github.com/jssjr/carbonate
• https://github.com/jjneely/buckytools
• https://github.com/dgryski/carbonmem
• https://github.com/grobian/carbonwriter
![Page 112: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/112.jpg)
Планы
• Патч statsd → ES
• Патч carbonserver → carbonlink
![Page 113: Мониторинг. Опять, rootconf 2016](https://reader031.vdocuments.pub/reader031/viewer/2022020214/58889ab31a28ab264b8b4c3b/html5/thumbnails/113.jpg)
feel free to ask• Vsevolod Polyakov
• skype: ctrlok1987
• github.com/ctrlok
• twitter.com/ctrlok
• slack: HangOps
• Gitter: dev_ua/devops
• skype: DevOps from Ukraine
• slack.ukrops.club
Мы хайрим!