Linker Networks Corp.http://www.linkernetworks.com

Use Raspberry Pi + Fluentd + GCP Cloud Logging, BigQuery

as IoT Data Collect & AnalysisSimon Su @ LinkerNetworks

var simon = {/** I am at GCPUG.TW **/};

simon.GDE = 'https://developers.google.com/experts/people/peihsin-su';

simon.nodejs = ‘http://opennodes.arecord.us';

simon.googleshare = 'http://gappsnews.blogspot.tw'

simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw';

simon.blog = ‘http://peihsinsu.blogspot.com';

simon.slideshare = ‘http://slideshare.net/peihsinsu/';

simon.email = ‘simonsu.mail@gmail.com’;

simon.say(‘Let’s go Cloud!');

Welcome to join us!

What we use...● Docker

● Fluentd

● Google Cloud Logging

● Google BigQuery

● SpreadSheet

Docker

● Build, Ship and Run any

App, Anywhere

● DevOps spirit

● Fast to scale

Fluentd

● Easy and fast● Message delivery for at

most once, at least once, exactly once

BigQuery

● TB Level data analysis

● Fast mining tool

● SQL like interface

● Multi-dataset join support

● Cheap & Pay by Use

Google BigQuery - Big data with SQL like query feature, but fast...

Cloud Logging

Architecture

● BigQuery: Data Warehouse and

support query and analysis

● Cloud Logging: Log gateway

and easy to transfer to one or

multi destination with guarantee

● Fluentd: Device layer gateway

and support multi source for

user to extend http(s) input

tcp/udp input

forward / secure forward

other inputs

● Customize schema● Co-relation● Higher quota and

performance

● Web console● Export available for BQ,

GCS, Pub/Sub● Configure input source

Prepare Log Gateway using Cloud Logging

Still no GCP project?? (https://cloud.google.com/free-trial/)

curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh

sudo bash install-logging-agent.sh

service google-fluentd start

Install Cloud Log AgentFrom: https://cloud.google.com/logging/docs/agent/installation

Configure # /etc/google-fluentd/google-fluentd.conf<source> type http port 9880</source><source> @type forward port 24224</source><filter *.**> @type stdout</filter># Do not collect fluentd's own logs to avoid infinite loops.<match fluent.**> type null</match># Configure all sources to output to Google Cloud Logging<match *.**> type google_cloud # Set the chunk limit conservatively to avoid exceeding the limit # of 10MB per write request. buffer_chunk_limit 2M flush_interval 5s # Never wait longer than 5 minutes between retries. max_retry_wait 300 # Disable the limit on the number of retries (retry forever). disable_retry_limit # Use multiple threads for processing. num_threads 8</match>

Set config to open fluentd forward and also screen output for debug..

Test run & watch log...

# service google-fluentd start

# tail -f /var/log/google-fluentd/google-fluentd.log

2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:

{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}

2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:

{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}

2016-09-02 09:46:44 +0000 hddusage.simon-project.a_54ERXdr4inKCGiAAAA:

{"catg":"hddusage","ctime":null,"size":null,"used":null,"avail":null,"use":null,"linker_devid":"simon-project.a_54ERXdr4inKCGiAAAA"}

….(skip)

Prepare a Docker enabled Pi

Install Docker into Fluentd

http://blog.hypriot.com/downloads/

Build the Fluentd Docker for Pi

Getting start with fluentd… (https://peihsinsu.gitbooks.io/fluentd-in-action/content/)

Build your Raspberry Pi Fluentd - Dockerfile

# DockerfileFROM hypriot/rpi-alpine-scratchMAINTAINER linkeriotRUN adduser -D -g '' -u 1000 dockerENV FLUENTD_VERSION 0.14RUN apk --update add \ build-base \ ca-certificates \ ruby-dev \ && \ rm -rf /var/cache/apk/* && \ echo 'gem: --no-document' >> /etc/gemrc && \ gem install fluentd -v $FLUENTD_VERSION && \ apk del build-baseWORKDIR /fluentdRUN mkdir log etc plugins config.dCOPY fluent.conf /fluentd/etc/ONBUILD COPY fluent.conf /fluentd/etc/ONBUILD COPY plugins/* /fluentd/plugins/ENV FLUENTD_OPT=""ENV FLUENTD_CONF="fluent.conf"EXPOSE 24224EXPOSE 8888VOLUME ["/fluentd/log"]CMD fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT

Build your Raspberry Pi Fluentd - fluentd.conf

# fluentd.conf@include /fluentd/config.d/*.conf<label @mainstream> <filter **> @type record_transformer <record> linker_devid "#{ENV['TAG']}" </record> </filter> <match **> @type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 16 hard_timeout 60s <server> name loggateway host "#{ENV['FDN']}" port "#{ENV['FPORT']}" weight 60 </server> <secondary> @type file path /var/log/fluent/forward-failed </secondary> </match></label>

Build and publish

● docker build -t your-image-name .● docker push your-image-name

Run the Docker...

Run from your images

docker run -d -p 24224:24224 --restart=always \ -e FDN=[remote-server-ip] \ -e FPORT=[remote-server-port] \ -e TAG=[your-tag] \ -v /var/log:/fluentd/log \ -v /data/cfg:/fluentd/config.d \ yout-image-name

Run using linker build images

docker run -d -p 24224:24224 --restart=always \ -e FDN=[remote-server-ip] \ -e FPORT=[remote-server-port] \ -e TAG=[your-tag] \ -v /var/log:/fluentd/log \ -v /data/cfg:/fluentd/config.d \ linkeriot/iot-log-agent

Set log forward to BigQuery

Check log insert status...

Enable log forward to BigQuery...

BigQuery from SpreadSheet

Try your query….

Create Apps Script from Google SpreadSheet

Using my sample...

Get source from gist: http://goo.gl/J90QD6

Test run the script

Setup the schedule...

Create the chart...

The EndQ&A