büyük veriyle büyük resmi görmek

Seeing the Big Picture Through Big Data

Onur Karadeli

Mustafa Murat Sever

March-2016C1 - Public

Agenda

• What is Big Data?• Use Cases• Apache Hadoop Ecosystem• Q&A

C1 - Public

Big Data ?

C1 - Public

What is Big Data ?

4C1 - Public

Big Data is growing (Google Trends)

5C1 - Public

Definition of Big Data

Big data is a term for data sets that are so large or complex that traditional data processing applications are inadequate.

Challenges include analysis, capture, data curation, search, sharing, storage, transfer, visualization, querying and information privacy.

-BigData WIKIPEDIA

What is Big ?

6C1 - Public

The ‘3V’ s

• Volume• Velocity• Variety

7C1 - Public

Volume

• %40 Growth per year

• 50 Zettabytes by 2020

Ref:Where-is-your-data-FINAL-5a

8C1 - Public

Volume Scalibility

Single TV

Multi TVs

9C1 - Public

Velocity

Ref: http://wersm.com/how-much-data-is-generated-every-minute-on-social-media/10C1 - Public

Velocity – Realtime triggers

11C1 - Public

Variety

12

Ref: Relational Solutions

C1 - Public

Additional ‘V’ s

• Veracity• Variability• Visualization• Value

Ref: http://blog.sqlauthority.com

Where to Use Big Data

C1 - Public

It’s a Big Family

15C1 - Public

16

Human Being as a Big Data Source

C1 - Public

Every moment, new data...

17C1 - Public

18

They are smart now

C1 - Public

Health

19https://www.youtube.com/watch?v=Lyv0_GIGSbY

C1 - Public

Some professions will disappear

News reporters Sonographers LawyersSports reporters Phlebotomists Compliance officers/workersWall street reporters Radiologists Bill collectorsJournalists Psychotherapists Meeting/event plannersAuthors Counselors Fitness coachesPsychologists Cost estimators CryptographersMilitary planners Accountants Financial planners/advisorsLogisticians Dietitians Tax advisors Interpreters/translatorsNutritionists Customer service reps AuditorsDoctors Teachers

* By Thomas Frey (Senior Futurist @ Da Vinci Institute)

C1 - Public

Not Only Humans ...

21C1 - Public

Not Only Humans: Connected Cows !

22C1 - Public

Not Only Humans: Connected Cows !

23C1 - Public

New opportunities for tech companies and new brands

24C1 - Public

Social Data

25C1 - Public

Visuality is important

26

https://www.youtube.com/watch?v=ujcrJZRSGkg

C1 - Public

Just Music

27C1 - Public

Not a rocket science, but ...

28

Discover Weekly Data Flow

C1 - Public

Not a rocket science, but ...

29

Implicit Matrix Factorization

C1 - Public

Other trend use cases

30C1 - Public

Apache Hadoop

C1 - Public

Apache Hadoop • Open-Source Projects/Sub-projects of

Apache.

• Core projectsHDFS: Hadoop Distributed File SystemMapReduce: Distributed Data processing

...• Hadoop is not a database.

• Move computation to data !

• Now- %32 percent of all enterprise uses Apache Hadoop.

32C1 - Public

Apache Hadoop History

• 2003 Google File system paper• 2006 Hadoop subproject created• 2008 Sort record: Running on a 910-node cluster, Hadoop sorted one

terabyte in 209 seconds• 2009 Yahoo runs 17 clusters with 24,000 machines• 2011 Facebook, LinkedIn, eBay and IBM collectively contribute 200,000

lines of code

Ref: https://en.wikipedia.org/wiki/Apache_Hadoop

33C1 - Public

Apache Hadoop Base Components & Enablers

Ref: http://synerzip.com - Innovation – It’s in our DNA

34C1 - Public

BI & Visualization example

35

Ref: http://forums.bsdinsight.com/articles/?page=4

C1 - Public

Hadoop Platforms

Ecosystem Management Software for Platform Management.

36

Examples:• Cloudera• Hortonworks• IBM• Pivotal

C1 - Public

HDFS File Storage Architecture

37C1 - Public

Hadoop Topology

38C1 - Public

Task Management

39C1 - Public

Map & Reduce

40C1 - Public

The Best Big Data Team should have ...

41

• Data Hygienists – for clean data

• Data Explorers – discover data to use

• Business Solution Architects – combine data for a use case

• Data Scientists – for the right model

• Campaign Expert – for the best benefit

* From HBR : https://hbr.org/2013/07/five-roles-you-need-on-your-bi

C1 - Public

Data Scientists Skills

42C1 - Public

Thank you

C1 - Public