big data in the cloud - solutions & apps

Big Data In The CloudGoogle Solutions Team

Speaker

Wally YauGoogle Cloud PlatformSolutions Architect

http://about.me/wally_yau

Google Solutions Team

Who Are We ?

● Experienced Solutions Architects● Seasoned Software Engineers

Google Solutions Team

What We Provide ?

● https://cloud.google.com/resources/

Solution PapersBest PracticesCase Studies

Sample Apps

Live Demos

BigQuery

● Familiar SQL Interface● Easy Web Interface● Public data set is available

○ Query 1 - Wikipedia Page View○ Query 2 - Compute Engine Instance○ Query 3 - Mother’s Age By State○ Query 4 - Mother’s Age By State/Region - Spreadsheet Integration

Solution Papers

Solution Papers

Getting Started With BigQuery https://cloud.google.com/resources/articles/getting-started-with-google-bigquery

Solution Papers

Manage Hadoop Clusters on Compute Enginehttps://cloud.google.com/resources/articles/managing-hadoop-clusters-on-google-compute-engine

Solution Papers

Apache Hadoop, Hive and Pig on Google Compute Enginehttps://cloud.google.com/resources/articles/apache-hadoop-hive-and-pig-on-google-compute-engine

Sample Apps

Sample Apps

Google Compute Engine Cluster For Hadoop

https://github.com/GoogleCloudPlatform/solutions-google-compute-engine-cluster-for-hadoop

Hadoop Master

Hadoop Worker

Hadoop Worker

Hadoop Worker

Data Files

MapperReducer

MapperReducer

MapperReducer

MapperReducer

Sample Apps

Automated File Loader for BigQuery

https://github.com/GoogleCloudPlatform/solutions-automated-file-loader-for-bigquery

Sample Apps

Using ETL Tool on Google Compute Engine

https://github.com/GoogleCloudPlatform/Solutions-Using-ETL-tool-on-Google-Compute-Engine

Demos

Data Sensing Lab

● http://data-sensing-lab.appspot.com/kiosks● End to end solution using Google Cloud Platform

Demos

Data Sensing Lab Architecture

Google App Engine Instances Compute

Engine Instances

lease Tasks from Pull Queues

Sample Apps

Data Pipeline

Feedbacks

Feedbacks