JSUG 勉強会〜 SpringOne Platform 2016 報告会　－　 Takuya Saeki

• 所属三井情報株式会社（ MKI ： Mitsui Knowledge Industry ）

R&D 部• 興味クラウドネイティブみたいなこと全般

佐伯 拓哉（ Takuya Saeki ）

自己紹介

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Data Microservices in the Cloud

By Mark Pollack

HTTP

JMS

Kafka

File

HDFS

Cassandra

HAWQ

JDBC

?

Real Time Analytics

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring XD Streams

Container Containergpfdist

Cassandrajms

http

ZooKeeper

Message Broker

XD Admin

stream1 = http | cassandrastream2 = jms | gpfdist

On Metal/VMs

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring XD Limitations• How to scale up/down instances at runtime?• How to upgrade/downgrade module instances at runtime?• How to specify resources unique to each module, e.g.

memory?• Container architecture lead to parent/child class loader issues• Too many libraries in root class path

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Refactoring to a Microservice Architecture

「コンテナに組み込まれた複数モジュール」から「単体で実行されるスタンドアロンアプリ」 へ

「独自ランタイム」 から「プラットフォーム委任」 へ

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Data Microservices• Stand-alone, production grade applications focused on data

processing• Communicating with ‘lightweight mechanisms’ – messaging

middleware• ‘Event Driven’ - Microservices“ 一つのことを行い、またそれをうまくやるプログラムを書け。”“ 協調して動くプログラムを書け。”“ 標準入出力（テキスト・ストリーム）を扱うプログラムを書け。標準入出力は普遍的インターフェースなのだ。”

$ cat book.txt | tr ' ' '\ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Application Types• Long lived Stream applications

• Spring Cloud Stream • Short lived Task Applications

• Spring Cloud Task

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Stream• Spring Boot based event-driven microservice framework• Opinionated primitives for streaming applications

• Persistent Publish/Subscribe semantics• Consumer Groups• Partitioning

• Pluggable messaging middleware bindings• Programming model focused on input/output objects

• Adaptable to different event processing APIs

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Stream Applications

Cassandra

java –jar cassandrasink-1.0.0.RELEASE.jar --ingestQuery=<some cql> --spring.cassandra.keyspace=tweetdata --spring.cloud.stream.bindings .input.destination=dataDest

http

java –jar twittersource-1.0.0.RELEASE.jar --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings .output.destination=dataDest

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Stream orchestration in Spring Cloud Data Flow

ingest = twitterstream | cassandraStream Definition

|Spring Cloud Stream ApplicationsMap DSL names to maven/docker artifacts

dataflow:>stream create --name ingest --definition “twitterstream | cassandra” --deploy

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Features: Persistent Messaging

HTTP log

s1.http DLQ

Message Broker

• Production Ready• RabbitMQ• Kafka

• Experimental • JMS• Google PubSub

• Planned • Kinesis

s1 = http | log

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Features: Named Destinations

HTTP

JMS

S3

myInputDestination

s1 = http > :myInputDestination

s2 = jms > :myInputDestination

s3 = aws-s3 > :myInputDestination

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Features: Consumer Groups

HTTP

s1.http

HDFS

s1 = http | hdfs

s2 = :s1.http > counter

COUNTER

group: s1 group: s2

HDFS

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Simple Real Time Analyticstweets = twitterstream | hdfs

analytics = :ingest.twitterstream > field-value-counter --fieldName=lang

HTTP

s1.http

HDFS COUNTER

Data Flow ServerREST API

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Stream Programming Model

@EnableBinding(Processor.class)public class TransformProcessor {

@StreamListener(“input”) @SendTo(“output”) public String transform(String s) { return s.toUpperCase(); }}

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Stream Programming Model

@EnableBinding(Processor.class)public class TransformProcessor {

@StreamListener @Output(“output”) public Flux<String> transform(@Input(“input”) Flux<String> input) { return input.map(s -> s.toUpperCase()); }}

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Stream Programming Model

@EnableBinding(Processor.class)public class TransformProcessor {

@StreamListener @Output(“output”) public Flux<WordCount> countWords(@Input("output") Flux<String> words) { return words.window(ofSeconds(5), ofSeconds(1)) .flatMap(window -> window.groupBy(word -> word) .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1) .map(count -> new WordCount(group.key(), count)))); }}

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Platform Runtimes

Docker Swarm Apache YARN

Apache Mesos + Marathon

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Data Flow Deployment Platforms

Data Flow ServerREST API

Deployer SPI

SCDF FloSCDF Shell

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Data Flow Streams

gpfdist

cassandrajms

http

stream1 = http | cassandrastream2 = jms | gpfdist

Message Broker

Data Flow Server DBPlatform Runtime

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Deployment: Partitioning and Instance Count

http

http

work

work

work

hdfs

hdfs

hdfs

hdfs

Load Balancer

stream create s1 --definition “http | work | hdfs”

stream deploy s1 --propertiesFile ingest.properties

app.http.count=2app.work.count=3app.hdfs.count=4app.http.producer.partitionKeyExpression=payload.id

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Deployment: Resource Management

http

http

work

work

work

app.work.spring.cloud.deployer.cloudfoundry.memory=2048

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Task• Spring Boot based framework for short lived processes

• Auto-configuration provides a task repository and pluggable data source

• Result of each process persists beyond the life of the task for future reporting

• Tasks can be any arbitrary short lived code• Well integrated with Spring Batch

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Task Orchestration in Spring Cloud Data Flow>task create jdbc2hdfs –sql=‘select * from table’>task launch jdbc2hdfs

jdbc2hdfs

Data Flow Server DB Task NameStart TimeEnd TimeExit CodeExit MessageLast Updated TimeParameters

task-event

Message Broker

job-execution-eventsstep-execution-

eventsitem-read-eventsitem-process-eventsitem-write-eventsskip-events

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Data Flow Tasks

spark

Data Flow Server DB

http | task-launcher

sqoop

Message Broker

task-event

Platform Runtime

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Spring Cloud Task Programming Model@SpringBootApplication@EnableTaskpublic class ExampleApplication {

@Bean public CommandLineRunner commandLineRunner() { return strings -> System.out.println("Executed at :" + new SimpleDateFormat().format(new Date())); }

public static void main(String[] args) { SpringApplication.run(ExampleApplication.class, args); }}

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Provided Applications

• ~60 stream and task apps• https://github.com/spring-cloud/spring-cloud-stream-app-starters• https://github.com/spring-cloud/spring-cloud-task-app-starters/

• Customize provided apps - http://start-scs.cfapps.io/ • Create new stream/task apps - http://start.spring.io/• Easy import of provided apps/tasks

• dataflow> app import --uri http://bit.ly/1-0-2-GA-stream-applications-kafka-maven

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

UI : Dashboard with Designer

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

XD to SCDF - Terminology

XD-Admin Data Flow Server(local, CF, YARN, k8s, Mesos)

XD-Container N/A

Modules Applications

Admin UI Dashboard

Message Bus Binders

Job Task

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

DEMO

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Upcoming features• Some ‘porting’ from XD

• Batch Job DSL + Designer• Role based access

• Looking forward• Spring Cloud Sleuth• JavaDSL• In-place application version upgrades with Spinnaker• Application Groups• Polyglot• Expanded analytics with Redis and Python/R ecosystem• More provided apps/tasks

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Related Talks• Building Resilient and Evolutionary Data Microservices – Tuesday

2:00pm• Cloud Native Java – Tuesday 2:00pm• Task Madness - Modern On Demand Processing – Tuesday 2:40pm• Spinnaker – Land of a 1000 Builds – Tuesday 5:00pm• Spring and Big Data – Tuesday 5:00pm• Migrating from Spring XD to Spring Cloud Data Flow – Thursday

10:10am• Orchestrate All the Things! with Spring Cloud Data Flow – Thursday

11:10am• Cloud Native Streaming and Event-Driven Microservices – Wednesday

4:20pm

Unless otherwise indicated, these s l ides are © 2013-2016 Pivota l Software , Inc . and l icensed under a Creative Commons Attr ibution-NonCommerc ia l l i cense:

http:/ /c reativecommons .org/ l icenses/by-nc/3.0/

Get Started…• http://cloud.spring.io/spring-cloud-dataflow/• http://cloud.spring.io/spring-cloud-stream/• http://cloud.spring.io/spring-cloud-task/• https://github.com/spring-cloud/spring-cloud-deployer