cassandra2013sparktalkfinal-130618122907-phpapp01

8/9/2019 cassandra2013sparktalkfinal-130618122907-phpapp01

1/128


2/128

Who is this guy

Sta!Engineer, Compute and Data Services, Ooyala

Building multiple web-scale real-time systems on top of C*, Kafka,Storm, etc.

Scala/Akka guy

Very excited by open source, big data projects - share some today

@evanfchan

Tuesday, June 18, 13


3/128

Agenda



4/128

Agenda

Ooyala and Cassandra



5/128

Agenda


What problem are we trying to solve?



6/128

Agenda



Spark and Shark



7/128

Agenda



Spark and Shark

Our Spark/Cassandra Architecture



8/128

Agenda



Spark and Shark

Our Spark/Cassandra Architecture

Demo



9/128

Cassandra at Ooyala

Who is Ooyala, and how we use Cassandra



10/128

OOYALAPowering personalizedvideo

experiences across all screens.



11/128

Founded in 2007

Commercially launch in 2009

230+ employees in Silicon Valley, LA, NYC,London, Paris, Tokyo, Sydney & Guadalajara

Global footprint, 200M unique users,110+ countries, and more than 6,000 websites

Over 1 billion videos played per monthand 2 billion analytic events per day

25% of U.S. online viewers watch videopowered by Ooyala

COMPANY OVERVIEW



12/128

TRUSTED VIDEO PARTNER



13/128

We are a large Cassandra user



14/128


11 clusters ranging in size from 3 to 36 nodes



15/128



Total of 28TB of data managed over ~85 nodes



16/128




Over 2 billion C* column writes per day



17/128





Powers all of our analytics infrastructure



18/128





Powers all of our analytics infrastructure

Much much bigger cluster coming soon



19/128

What problem are we trying tosolve?

Lots of data, complex queries, answered really quickly... but how??



20/128

From mountains of useless data...



21/128

To nuggets of truth...



22/128


Quickly



23/128


Quickly

Painlessly



24/128


Quickly

Painlessly

At scale?



25/128

Today: Precomputed aggregates



26/128


Video metrics computed along several high cardinality dimensions



27/128



Very fast lookups, but inflexible, and hard to change



28/128




Most computed aggregates are never read



29/128




Most computed aggregates are never read

What if we need more dynamic queries?

Top content for mobile users in France

Engagement curves for users who watched recommendations Data mining, trends, machine learning



30/128

The static - dynamic continuum

Super fast lookups

Inflexible, wasteful

Best for 80% mostcommon queries

Always compute resultsfrom raw data

Flexible but slow

100% Precomputation 100% Dynamic



31/128

The static - dynamic continuum

Super fast lookups

Inflexible, wasteful

Best for 80% mostcommon queries

Always compute resultsfrom raw data

Flexible but slow

100% Precomputation100% Dynamic



32/128

Where we want to be

Partly dynamic

Pre-aggregate mostcommon queries

Flexible, fast dynamicqueries

Easily generate manymaterialized views



33/128

Industry Trends



34/128

Industry Trends

Fast execution frameworks

Impala



35/128

Industry Trends


Impala

In-memory databases

VoltDB, Druid



36/128

Industry Trends


Impala

In-memory databases

VoltDB, Druid

Streaming and real-time



37/128

Industry Trends


Impala

In-memory databases

VoltDB, Druid

Streaming and real-time

Higher-level, productive data frameworks Cascading, Hive, Pig



38/128

Why Spark and Shark?

Lightning-fast in-memory cluster computing



39/128

Introduction to Spark



40/128


In-memory distributed computing framework



41/128



Created by UC Berkeley AMP Lab in 2010



42/128




Targeted problems that MR is bad at:

Iterative algorithms (machine learning)

Interactive data mining



43/128







More general purpose than Hadoop MR



44/128







More general purpose than Hadoop MR

Active contributions from ~ 15 companies



45/128

HDFS

Map

Reduce

Map

Reduce



46/128

HDFS

Map

Reduce

Map

Reduce



47/128

HDFS

Map

Reduce

Map

Reduce

Data Source

map()

join()

Sour



48/128

HDFS

Map

Reduce

Map

Reduce

Data Source

map()

join()

Sour

cache()



49/128

HDFS

Map

Reduce

Map

Reduce

Data Source

map()

join()

Sour

cache()

transform



50/128

Throughput: Memory is king

6-node C*/DSE 1.1.9 cluster,

Spark 0.7.0



51/128


0 37500 75000 112500 15000

C*, cold cache

C*, warm cache

Spark RDD


Spark 0.7.0



52/128


0 37500 75000 112500 15000

C*, cold cache

C*, warm cache

Spark RDD


Spark 0.7.0



53/128


0 37500 75000 112500 15000

C*, cold cache

C*, warm cache

Spark RDD


Spark 0.7.0



54/128


0 37500 75000 112500 15000

C*, cold cache

C*, warm cache

Spark RDD


Spark 0.7.0



55/128

Developers love it



56/128

Developers love it

I wrote my first aggregation job in 30 minutes



57/128

Developers love it


High level distributed collections API



58/128

Developers love it



No Hadoop cruft



59/128

Developers love it



No Hadoop cruft

Full power of Scala, Java, Python



60/128

Developers love it



No Hadoop cruft


Interactive REPL shell



61/128

Developers love it



No Hadoop cruft



EASY testing!!



62/128

Developers love it



No Hadoop cruft



EASY testing!!

Low latency - quick development cycles



63/128

Spark word count example

1packageorg.myorg;

2

3importjava.io.IOException;

4importjava.util.*;

5

6importorg.apache.hadoop.fs.Path;

7importorg.apache.hadoop.conf.*;

8importorg.apache.hadoop.io.*;

9importorg.apache.hadoop.mapreduce.*;

10importorg.apache.hadoop.mapreduce.lib.input.FileInputFormat;

11importorg.apache.hadoop.mapreduce.lib.input.TextInputFormat;

12importorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

13importorg.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

14

15publicclassWordCount {

16

17 publicstaticclassMap extendsMapper


64/128

Spark word count example

file = spark.textFile("hdfs://...")

file.flatMap(line => line.split(" "))

.map(word => (word, 1))

.reduceByKey(_ + _)

1packageorg.myorg;

2

3importjava.io.IOException;

4importjava.util.*;

5

6importorg.apache.hadoop.fs.Path;

7importorg.apache.hadoop.conf.*;

8importorg.apache.hadoop.io.*;

9importorg.apache.hadoop.mapreduce.*;

10importorg.apache.hadoop.mapreduce.lib.input.FileInputFormat;

11importorg.apache.hadoop.mapreduce.lib.input.TextInputFormat;

12importorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

13importorg.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

14

15publicclassWordCount {

16

17 publicstaticclassMap extendsMapper


65/128

The Spark Ecosystem

Spark

Tachyon- in-memory caching DFS



66/128

The Spark Ecosystem

Bagel-Pregel onSpark

Spark




67/128

The Spark Ecosystem


HIVE on Spark

Spark




68/128

The Spark Ecosystem


HIVE on Spark

Spark Streaming-discretized streamprocessing

Spark




69/128

Shark - HIVE on Spark



70/128


100% HiveQL compatible



71/128



10-100x faster than HIVE, answers in seconds



72/128




Reuse UDFs, SerDes, StorageHandlers



73/128





Can use DSE / CassandraFS for Metastore



74/128





Can use DSE / CassandraFS for Metastore

Easy Scala/Java integration via Spark - easier thanwriting UDFs



75/128

Our new analytics architecture

How we integrate Cassandra and Spark/Shark



76/128

From raw events to fast queries

RawEvents

RawEvents

RawEvents



77/128


IngestionC*

eventstore

RawEvents

RawEvents

RawEvents



78/128


IngestionC*

eventstore

RawEvents

RawEvents

RawEvents

Spark

Spark

Spark

View 1

View 2

View 3



79/128


IngestionC*

eventstore

RawEvents

RawEvents

RawEvents

Spark

Spark

Spark

View 1

View 2

View 3

SparkPq



80/128


IngestionC*

eventstore

RawEvents

RawEvents

RawEvents

Spark

Spark

Spark

View 1

View 2

View 3

Spark

Shark

Pq

AH



81/128

Our Spark/Shark/Cassandra Stack

Node1

Cassandra

Node2

Cassandra

Nod

Cassa



82/128


Node1

Cassandra

InputFormat

SerDe

Node2

Cassandra

InputFormat

SerDe

Nod

Cassa

InputF



83/128


Node1

Cassandra

InputFormat

SerDe

SparkWorker

Shark

Node2

Cassandra

InputFormat

SerDe

SparkWorker

Shark

Nod

Cassa

InputF

SparkWorker



84/128


Node1

Cassandra

InputFormat

SerDe

SparkWorker

Shark

Node2

Cassandra

InputFormat

SerDe

SparkWorker

Shark

Nod

Cassa

InputF

SparkWorker

Spark Master



85/128


Node1

Cassandra

InputFormat

SerDe

SparkWorker

Shark

Node2

Cassandra

InputFormat

SerDe

SparkWorker

Shark

Nod

Cassa

InputF

SparkWorker

Spark Master Job Server



86/128

Event Store Cassandra schema

t0 t1 t2 t3 t4

2013-04-05T00:00Z#id1

{event0:

a0}

{event1:

a1}

{event2:

a2}

{event3:

a3}

{even

a4}

Event CF



87/128

Event Store Cassandra schema

t0 t1 t2 t3 t4

2013-04-05T00:00Z#id1

{event0:

a0}

{event1:

a1}

{event2:

a2}

{event3:

a3}

{even

a4}

ipaddr:10.20.30.40:t1 videoId:45678:t1 providerId:50

2013-04-05T00:00Z#id1

Event CF

EventAttr CF



88/128

Unpacking raw events

t0 t12013-04-05T00:00Z#id1

{video: 10,

type:5}

{video: 11,

type:1}

2013-04-05T00:00Z#id2

{video: 20,

type:5}

{video: 25,

type:9}

UserID Video Tyid1 10 5



89/128


t0 t12013-04-05T00:00Z#id1

{video: 10,

type:5}

{video: 11,

type:1}

2013-04-05T00:00Z#id2

{video: 20,

type:5}

{video: 25,

type:9}


id1 11 1



90/128


t0 t12013-04-05T00:00Z#id1

{video: 10,

type:5}

{video: 11,

type:1}

2013-04-05T00:00Z#id2

{video: 20,

type:5}

{video: 25,

type:9}


id1 11 1

id2 20 5



91/128


t0 t12013-04-05T00:00Z#id1

{video: 10,

type:5}

{video: 11,

type:1}

2013-04-05T00:00Z#id2

{video: 20,

type:5}

{video: 25,

type:9}


id1 11 1

id2 20 5

id2 25 9



92/128

Tips for InputFormat Development



93/128


Know which target platforms you are developing for

Which API to write against? New? Old? Both?



94/128




Be prepared to spend time tuning your split computation

Low latency jobs require fast splits



95/128






Consider sorting row keys by token for data locality



96/128






Consider sorting row keys by token for data locality

Implement predicate pushdown for HIVE SerDes

Use your indexes to reduce size of dataset



97/128

Example: OLAP processing

t0

2013-04

-05T00:

00Z#id1

{video:

10,

type:5}

2013-04

-05T00:00Z#id2

{video:

20,type:5}

C* events



98/128


t0

2013-04

-05T00:

00Z#id1

{video:

10,

type:5}

2013-04

-05T00:00Z#id2

{video:

20,type:5}

C* events

OLAPAggregates

OLAPAggregates

OLAPAggregates

Cached Materialized Views

Spark

Spark

Spark



99/128


t0

2013-04

-05T00:

00Z#id1

{video:

10,

type:5}

2013-04

-05T00:00Z#id2

{video:

20,type:5}

C* events

OLAPAggregates

OLAPAggregates

OLAPAggregates


Spark

Spark

Spark

Union



100/128


t0

2013-04

-05T00:

00Z#id1

{video:

10,

type:5}

2013-04

-05T00:00Z#id2

{video:

20,type:5}

C* events

OLAPAggregates

OLAPAggregates

OLAPAggregates


Spark

Spark

Spark

Union

Query 1:by Provid



101/128


t0

2013-04

-05T00:

00Z#id1

{video:

10,

type:5}

2013-04

-05T00:00Z#id2

{video:

20,type:5}

C* events

OLAPAggregates

OLAPAggregates

OLAPAggregates


Spark

Spark

Spark

Union

Query 1:by Provid

Query 2:

content mobile



102/128

Performance numbers


Spark 0.7.0



103/128

Performance numbers

Spark: C* -> OLAP aggregatescold cache, 1.4 million events

130 seconds


Spark 0.7.0



104/128

Performance numbers


130 seconds

C* -> OLAP aggregateswarmed cache

20-30 seconds


Spark 0.7.0



105/128

Performance numbers


130 seconds

C* -> OLAP aggregateswarmed cache

20-30 seconds

OLAP aggregate query via Spark

(56k records)

60 ms


Spark 0.7.0



106/128

OLAP WorkFlow

Aggregation JobSparkExecutors

REST Job Server

Aggregate



107/128

OLAP WorkFlow

Aggregation JobSparkExecutors

Cassandra

REST Job Server

Aggregate



108/128

OLAP WorkFlow

DatasetAggregation JobSparkExecutors

Cassandra

REST Job Server

Aggregate



109/128

OLAP WorkFlow

DatasetAggregation Job Query JobSparkExecutors

Cassandra

REST Job Server

Aggregate Query



110/128

OLAP WorkFlow


Cassandra

REST Job Server

Aggregate Query

Result



111/128

OLAP WorkFlow


Cassandra

REST Job Server

Query Job

Aggregate Query

Result

Query

Result



112/128

Fault Tolerance



113/128

Fault Tolerance

Cached dataset lives in Java Heap only - what if process dies?



114/128

Fault Tolerance


Spark lineage - automatic recomputation from source, but this isexpensive!



115/128

Fault Tolerance



Can also replicate cached dataset to survive single node failures



116/128

Fault Tolerance




Persist materialized views back to C*, then load into cache -- nowrecovery path is much faster



117/128

Fault Tolerance




Persist materialized views back to C*, then load into cache -- nowrecovery path is much faster

Persistence also enables multiple processes to hold cached dataset



118/128

Demo time



119/128

Shark Demo

Local shark node, 1 core, MBP

How to create a table from C* using our inputformat

Creating a cached Shark table

Running fast queries



120/128

Creating a Shark Table from InputFormat



121/128

Creating a cached table


Querying cached table


122/128



123/128

THANK YOU



124/128

THANK YOU

@evanfchan



125/128

THANK YOU

@evanfchan

[email protected]

mailto:[email protected]:[email protected]


126/128

THANK YOU

@evanfchan

[email protected]



127/128

THANK YOU

@evanfchan

[email protected]

WE ARE HIRING!!



128/128

Spark: Under the hood

Map DatasetReduce Map

Driver Map DatasetReduce Map

Map DatasetReduce Map

One executor process per node

cassandra2013sparktalkfinal-130618122907-phpapp01

Documents