redis acc
DESCRIPTION
TRANSCRIPT
REDIS Management & Cell Architecture
Redis.io
Redis/Twemproxy Contributor
카카오 홈 개발자
Agenda REDIS Management REDIS HA CELL Architecture
Agenda REDIS Management REDIS HA CELL Architecture
Redis Single Threaded In Memory
Persistent Replication
Collections
Redis is Single Threaded!
Single Thread Means!
Don’t execute long task.
Example: Keys command Flushall/flushdb command
O(n)
Keys *
di = dictGetSafeIterator(c->db->dict); allkeys = (pattern[0] == '*' && pattern[1] == '\0'); while((de = dictNext(di)) != NULL) { …… stringmatchlen(pattern,plen,key,sdslen(key),0) }
Memcache’s flush is faster than Redis’s flush.
Memcache’s flush is faster than
Redis’s flush?
FlushAll-Redis for (i = 0; i < ht->size && ht->used > 0; i++) { dictEntry *he, *nextHe; if ((he = ht->table[i]) == NULL) continue; while(he) { nextHe = he->next; dictFreeKey(d, he); dictFreeVal(d, he); zfree(he); ht->used--; he = nextHe; } }
FlushAll-Memcache
if (exptime > 0) settings.oldest_live = realtime(exptime) - 1; else /* exptime == 0 */ settings.oldest_live = current_time - 1;
FlushAll-Memcache if (settings.oldest_live != 0 && settings.oldest_live <= current_time && it->time <= settings.oldest_live) { do_item_unlink(it, hv); do_item_remove(it); it = NULL; }
FlushAll
Cache Item Count Time
Memcache 1,000,000 1~2ms
Redis 1,000,000 1000ms(1 second)
Redis Single Threaded In Memory
Persistent Replication
Collections
In Memory Fast but can’t store big data. 32bit: Maximum 3G Data in Memory 64bit: No Max memory. using swap memory in OS.
Redis Single Threaded In Memory
Persistent Replication
Collections
Collections Memcached supports just K-V List Set Sorted Set Hash
Don’t insert too many items into one collections.
Delete collections
Item Count Time
list 1,000,000 1000ms(1 second)
set
Sorted set
hash
Redis Single Threaded In Memory
Persistent Replication
Collections
Redis can make its memory snapshot.
RDB is not Relation DBMS.
RDB is redis snapshot name.
Fork()
RDB - BAD • Bad Performance in Large memory.
• Twice memory problem.
• Denying write problem when storing RDB fails.
• If you just want Cache. Turn off RDB
RDB – Large Memory • Performance is relevant to Memory
Size.
17.1 GB 34.2 GB
68.4 GB
117 GB
RDB – Twice Memory • fork() and COW(copy on write) Issue
–In Write Heavy System:
RDB – Twice Memory
RDB – Twice Memory
RDB – Twice Memory
Real Case Study • Background
–Can’t write to Redis Server
–Sentinel doesn’t find Server’s failure.
Real Case Study • Reason –If redis fails to save RDB, Redis basically
denies write operations from client.
–“MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
Real Case Study • Reason if (server.stop_writes_on_bgsave_err &&
server.saveparamslen > 0 && server.lastbgsave_status == REDIS_ERR && c->cmd->flags & REDIS_CMD_WRITE) { flagTransaction(c); addReply(c, shared.bgsaveerr); return REDIS_OK; }
Real Case Study • Solution #1
• Solution #2
config set stop-writes-on-bgsave-error no
Turn off RDB Setting
AOF
AOF is append only file.
AOF memorizes all requests.
AOF *3\r\n$3\r\nset\r\n$1\r\nA\r\n$3\r\n123\r\n*3\r\n$3\r\nset\r\n$1\r\nB\r\n$3\r\n123\r\n*3\r\n$3\r\nset\r\n$1\r\nC\r\n$3\r\n123\r\n
AOF *3 $3 Set $1 A ……
AOF Rewrite When size of AOF grows than redis rewrite memory state to AOF.
Redis Single Threaded In Memory
Persistent Replication
Collections
Redis support master/slave replication
Replication •Support Chained Replication
Master 1st Slave 2nd Slave
1st slave is master of 2nd slave
Replication
Master Slave replicationCron
Health check
Replication
Master Slave replicationCron
Health check
Replication
Master Slave replicationCron
When master reruns, Resync with Master
Mistake: Replication
Master Slave replicationCron
Slave will has no data after resyncing
If master has no data.
Replication Don’t forget “slave of no one”
When Redis forks… 1. BGSAVE(RDB) 2. AOF Rewrite(AOF) 3. Replication(RDB) even though, you turned it off
Agenda REDIS Management REDIS HA CELL Architecture
Redis Sentinel
What Redis Sentinel
do?
1. Check Redis Liveness
1. Check Redis Liveness -Subjective Down -Objective Down
2. Choose good slave.
Good Slave NOT SDOWN, ODOWN, DISCONNECTED
NOT DEMOTE
Ping reply > info_validity_time
Slave_priority != 0
Info reply > info_validity_time
Choose new master
Sort Slave Priority
Runid
3. Promote it
Promoting Send Slaveof no one to new master
Send Slaveof [new master ip] [addr] to Other redis
Notify new master to clietns +switch-master
Set DEMOTE mark to old-Master
Is Sentinel Really Good?
Not Mature #1
Sentinel Conf port 26379 sentinel monitor mymaster 127.0.0.1 6379 2 sentinel down-after-milliseconds mymaster 2000 sentinel can-failover mymaster yes sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 900000
Quorum Count sentinel monitor mymaster 127.0.0.1 6379 2
We should check this count.
Sentinel Conf sentinel down-after-milliseconds mymaster 2000
If this value is Too small, It can cause some trouble.
-sdown/+sdown loop
Compare with Zookeeper Conf tickTime=2000 dataDir=/var/zookeeper clientPort=2181 initLimit=5 syncLimit=2 server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
Not Mature #2
If redis is down, when redis is loading data(RDB,
AOF), sentinel can’t register redis.
Even though, Sentinel is not mature. It is useful.
Redis Failover
Twemproxy
Twemproxy Redis
Redis
Redis
Client
Client
Client
Client
Client
Client
hashing
Twemproxy Pros
One connection per server Zero Copy Multi Hashing Algorithm Support Nickname
Twemproxy Cons
Not Support Full Command of Redis(Pub/Sub or MULTI)
Not Support HA
Twemproxy
Redis
Redis
Redis
Twemproxy
Twemproxy
L4
Redis-Sentinel TwemProxy Agent
http://www.codeproject.com/Articles/656965/Redis-Sentinel-TwemProxy-Agent
HA Proxy
HA Proxy
VRRP Twemproxy
Twemproxy
Twemproxy
Redis Master
Redis Slave
Redis Slave
Shard
Redis Master
Redis Slave
Redis Slave
Shard Sentinel
Agent Pub/Sub
Restart
Sharding
Sharding
WEB/AS
Master Master Master User A Data User C Data User D Data
User B Data User X Data User Z Data
User Y Data User E Data User F Data
Cell Architecture
Cell 0 Cell 1 Cell 2
User Cell Info Server
ID: CharSyam
CellID: 1, Status: Normal
Get/set
Cell Architecture
A Cell is Full-Set Can serve Users
Cell Examples #1
WEB/AS
Master Slave
WEB/AS WEB/AS
Cell Examples #2
WEB/AS WEB/AS WEB/AS
Master
Slave Slave Slave
READ only WRITE only
Failure of Cell Architecture
Cell 0 Cell 1 Cell 2
User Cell Info Server
ID: CharSyam
CellID: 1, Status: Normal
Can’t Service
Failure of Cell Architecture
Cell 0 Cell 1 Cell 2
User Cell Info Server
ID: CharSyam
Can’t response
Get/set
• Benefits
–Easy to extend
–Failure is limited to some Users in same cell.
–Can deploy specific feature to some cell users.
• Liabilities
–To need more servers.
• To build full-set
Cell Architecture
Q&A
Thank you