akka cluster and auto-scaling
Post on 20-Mar-2017
434 Views
Preview:
TRANSCRIPT
Akka Cluster andAuto-scaling
Ikuo MatsumuraCyberAgent, Inc.
2017/02/26
Akka Cluster
⾮中央集権的なノード群構築を⾏うAkka拡張 10ヶ⽉程運⽤してきた中からつまづいた点・学んだ点を紹介
• Decentralized cluster membership service
• no single point of failure, bottleneck
• distribute actors over multiple JVMs
• Applied to build a sub-system on AD serving
• Tens of servers
• Operations about 10 months
Requirements in our case
• Host a lot of Entity with low cost
• Fit existing Akka application
• Down-time is acceptable to some extent*
Akkaベースで多数のEntityを低コストで配備したい 多少のダウンタイムは許容できる
*online machine learning
Our application of Akka Cluster
永続ActorをCluster Shardingで配備 データ保管にコモディティサービスを使⽤
frontend frontend frontend
entitiesentities
…
…entities
frontend
• Existing app• Tens of nodes• Auto-scaling
• New sub-system• Several nodes
ElastiCache(Journal)
S3(Snapshot)
data stores
…
Challenges
• Strategy on unreachables removal
• Lifecycle of journals
運⽤する中でつまづいた2つの課題についてお話します
Membership Lifecycle in Cluster Specification
クラスタメンバーのライフサイクルの概観
http://doc.akka.io/docs/akka/2.4/common/cluster.html#Membership_Lifecycle
joining
up
downremoved
join
leaving
exiting
unreachable
leave
Joinning and Leader Action
“leader action”を経て、他メンバと通信できるようになる
joining
up
down
join
unreachable
Joinning and Leader Action
“leader action”を経て、他メンバと通信できるようになる
joining
up
down
unreachable
leader action
Joinning and Leader Action
Scale-in発⽣時、unreachable のままになる
joining
up
down
unreachable
failure detector
leader action
Joinning and Leader Action
leader actionが⾏えなくなる。 結果、新規メンバが他のメンバと通信できないままに。
joining
up
down
unreachable
leader action
Joinning and Leader Action
Scale-inをトリガにしたdown指定が必要
joining
up
down
unreachable
leader action
scale-in
trigger
mark as down
Joinning and Leader Action
unreachableを除くことでleader actionが再開可能に
joining
up
down
unreachable
leader action
Leader actions blocked by unreachables
leader actionが⾏えない状態のログの例
Members that are “up” but have not seen the current state
“Leader can currently not perform its duties”
Causes and actions on unreachables
Type of failures ExamplePossible
external action
network partitions -wait for recovery or
abandon a part
machine crashesscale-in mark as down
quarantined in akka remote layer
restart an actor system
unresponseive process
long GC restart a JVM
CPU starvation by credit shortage in EC2
re-create an instance
failure detector はエラーの原因までは区別できない 原因に応じてクラスタ外部からの回復措置が要る
Split Brain Resolver* (commercial add-on)
• Mark members as “down” when a part of the cluster become unreachable for some time
• Strategies
• Static Quorum
• Keep Majority - default in Lagom
• Keep Oldest
• Keep Referee
* http://doc.akka.io/docs/akka/rp-current/scala/split-brain-resolver.html
商⽤add-onである程度包括的に⾃動のdown指定が可能 ⼀定時間メンバの状態・到達可能性に変化がない時に発動
Reset cluster membership (poor man’s)
存命ノードを新しいクラスタに参加させ直す
seed(s)
old cluster (ddata) new cluster (ddata)
Reset cluster membership (poor man’s)
存命ノードを新しいクラスタに参加させ直す
seed(s)
old cluster (ddata) new cluster (ddata)
Reset cluster membership (poor man’s)
存命ノードを新しいクラスタに参加させ直す
seed(s)
old cluster (ddata) new cluster (ddata)
Caution
• Side-effect caused by app restart
• ddata is experimental (at Akka 2.4)
• Use Akka 2.4.8 or higher*
再起動による副作⽤やAkkaのバージョンに注意が必要
*has a fix on distributed pub-sub akka#20847
To keep cluster membership healthy
1. Trigger mark-as-down (or leave) on scale-in
2. Automate restart/recreation of AcotrSystem, JVM, server instance
3. Setup a fallback mechanism such as split brain resolver, or rejoining into a new cluster
unreachable対策のまとめ
Challenges
• Strategy of unreachables removal
• Lifecycle of journals
次に、2つ⽬の課題についてお話しします。
Journal
entitiesentities
…entities
ElastiCache(Journal)
S3(Snapshot)
Event Sourcingにおけるイベントストアに対応するAPI Journalをキャッシュのように運⽤する想定をした
Cleanup old journals in Redis plug-in*
JournalのDeleteMessageでは⼀部データが残るケースがある snapshotとのsequenceNrの⼀貫性に注意
key in Redis removed on deleteMessages
journal:$persistenceId Yes
journal:$persistenceId.highestSequenceNr No
* https://github.com/hootsuite/akka-persistence-redis/blob/master/src/main/scala/com/hootsuite/akka/persistence/redis/journal/RedisJournal.scala
Deleting highstSequenceNr could cause loading old version of snapshot.→ Keep only the latest snapshot.
Event Sourcing and Ecosystem
“it stores a complete history of the events associated with the aggregates in your domain”
Reference 3: Introducing Event Sourcing, CQRS Journey[CQJ]
本来のイベントストアはイベントの完全な履歴を持つ想定 そこから逸れるとエコシステム(plug-in)のサポートも弱くなる
Summary
• Lessons learned from devops of an Akka Cluster app
• Strategy on unreachables removal
• scale-in trigger
• automatic restart/recreation
• fallback mechanism; split-brain resolver / rejoining
• Lifecycle of journals
• cost of deviation from Event Sourcing
unreachableメンバを取り除く仕組みを各種⼊れておく Journalのキャッシュ的運⽤は意外と⼤変（なことがある）
Reference
[CQJ] Exploring CQRS and Event Sourcing, Dominic Betts, Julian Dominguez, Grigori Melnik, Fernando Simonazzi, Mani Subramanian, 2012, https://msdn.microsoft.com/en-us/library/jj554200.aspx
[PSE] Persistence - Schema Evolution, Akka Documentation, http://doc.akka.io/docs/akka/2.4/scala/persistence-schema-evolution.html
top related