Андрей Годин - Базы данных: Документоориентированная...
TRANSCRIPT
1
2
3
Mongodb, UGC project
-‐ billions objects; -‐ terabytes of data MongoDB; -‐ high Disk I/O; -‐ big size Indexes -‐ balancing & connec?ons to DB; -‐ troubleshoo?ng mongodb;
4
диск
~6,4 billion objects:
5
диск
~6 Tb filesize:
6
диск
~1.5 Tb IndexSize;
7
диск > 6 billion opera?ons every day;
8
диск -‐ over 50000 chunks
9
диск
Notes: -‐ write global lock to DB; -‐ no transac?ons; -‐ single threaded chunk moving job; -‐ 20000 connec?ons limit to mongod/mongos; -‐ Mongos unable to work in RO mode; -‐ no connec?on-‐pool
10
11
12
13
14
DBA at work:
15
Usual RS:
Secondary
Secondary
Secondary hidden
Primary
ReplicaSet:
UP UP
UP
UP
UP delay
16
Clean hidden and Startup:
Secondary
Secondary
Secondary hidden
Primary
ReplicaSet:
Startup
UP
UP
UP
17
Recover Secondary:
Secondary
Secondary
Secondary hidden
Primary
ReplicaSet:
DOWN
DOWN
UP
UP
rsync
18
Chunk-‐defrag RS:
Secondary
Secondary
Secondary hidden
Primary
ReplicaSet:
UP UP
UP
Defrag UP
Defrag UP delay
19
Recover Secondary:
Secondary Secondary
Secondary hidden
Primary
ReplicaSet:
DOWN
DOWN
UP
Defrag UP
rsync
20
Usual RS:
Secondary
Secondary
Secondary hidden
Primary
ReplicaSet:
UP UP UP defrag
UP defrag
UP defrag
21
Disk I/O:
-‐ ext4 -‐ raid0 и raid10 -‐ mount op?ons -‐ dirty pages -‐ SSD
22
Monitoring:
-‐ db.serverStatus() -‐ rs-‐indexes-‐consistency -‐ indexes-‐miss-‐ra?o -‐ backgroundFlushing -‐ heap_usage_bytes -‐ connec?ons -‐ replica?on -‐ qr|qw
23
mongostat -‐-‐discover:
24
Balancing:
-‐ sh.getBalancerState() -‐ moveChunk
25
Mongos:
26
Mongos drop tcp cfg:
drop tcp
27
Mongos drop tcp shard:
drop tcp
28
Mongos fast fix:
drop tcp
-‐j REJECT -‐-‐reject-‐with tcp-‐reset
29
Keepalived config:
30
Mongos RS down — ALL DOWN:
31
Thank you! Ques?ons?
Andrey Godin h=p://yandex.ru @airesp_ya agodin@yandex-‐team.ru