Reactive Elasticsearchwith Akka Streams

第二十回 #渋谷java

Naoki Takezoe@takezoen

BizReach, Inc

Motivation

● Real-time indexing in Elasticsearch● Resiliece of indexing process● Tuning indexing load automatically

Basic structure

Source Flow Sink

Read Write

Pull from down stream

Push from upstream

Akka Streams

● An implementation of Reactive Streams based on Scala and Akka○ Java API is also available

● Akka HTTP○ Covers HTTP server and client applications

● Alpakka○ Provides various Akka Streams connectors

Example

// Sourceval source = Source(1 to 100)

// Sinkval sink = Sink.foreach { x: Int => println(x) }

// Just connect Source and Sink directlysource.runWith(sink)

// Put Flow between Source and Sinksource.map(x => x * 2).runWith(sink)

Elasticsearch connector is availavle in Alpakka!

ElasticsearchSource

val source = ElasticsearchSource.typed[Book]( "source", // Index name "book", // Type name """{"match_all": {}}""", // Query ElasticsearchSourceSettings(5) // Buffer size)

Implementation of ElasticsearchSource

● Scroll Elasticsearch and buffer documents● Push docs when pulled from down stream● Read next window if read buffer is empty

ElasticsearchSink

val sink = ElasticsearchSink.typed[Book]( "sink", // Index name "book", // Type name ElasticsearchSinkSettings(5) // Buffer size)

Implementation of ElasticsearchSink

● Pull until buffer is full if buffer isn't full● Indexing by bulk request buffered docs● Run these processes in parallel

Demo

See testcase to know more details:

https://github.com/akka/alpakka/blob/master/elasticsearch/src/test/scala/akka/stream/alpakka/elasticsearch/ElasticsearchSpec.scala

TODO

● Automatic pull amount tuning○ By Looking up to metrics of Elasticsearch cluster

● Partial retry○ Retry failed documents when bulk request is failed

partially● Sliced scroll

○ Elasticsearch 5.x supports sliced scroll that makes possible to scroll in parallel