hive tools in nhn japan #hadoopreading

Hive Tools in NHN JapanHadoop Source Code Reading Vol.92012/05/30@tagomoris (TAGOMORI Satoshi)

12年5月30日水曜日

@tagomorisNHN Japan Corp

Web Service Division


Hive in NHN Japan

Reporting of access log (not analysis)

Pageviews and/or Unique Users?

Accesses under specified condition?

Hey, what numbers of accesses for our new features?

new bot accesses? any troubles?


SELECT yyyymmdd, count(is_pc(pa)) as pc, count(is_smartphone(pa)) as smartphone, count(is_mobilephone(pa)) as mobilephoneFROM ( SELECT yyyymmdd, parse_agent(agent) as pa FROM access_log WHERE service='__SERVICE__' AND (yyyymmdd='__1DAYS_AGO__' OR yyyymmdd='__2DAYS_AGO__') AND NOT flag ) xGROUP BY yyyymmddORDER BY yyyymmdd LIMIT 2


Today's topic

For Fluentd,See 'Software Design'

2012/06


Hadoop / HDFS

FluentdCluster

stream

Hoop Server (HttpFs)

stream

backup stream

realtime monitoring

Hive Server

Users(Web Browser)

Shib(Hive Client Web Application)

ShibUI(Query Management System)


Why Hive?Handmade MapReduce: Noooooooooooooooo

Pig? Hive?

All we loves 'xQL' like 'SQL'...

FORCE to throw away all queries

"処理を書き捨てる勇気"

We are likely to maintain 'programs' (like pig script)

With chainging data, BAD to maintain how to handle data


Client Tools?

'hive' command sucks

Hue (Beeswax for Hive)？

we want end-users to run 'SELECT' only.

we want HTTP API to work with another systems

Periodic query execution, and graph plotting

Miscellaneous extensions we want (and ease to write)


Copy&Paste Based Query Management

Non-refered QueriesMUST DIE


Hadoop / HDFS

FluentdCluster

stream


stream

backup stream

realtime monitoring

Hive Server

Users(Web Browser)




Shibhttps://github.com/tagomoris/shib

Hive Client Web Application

Run 'SELECT' queries only

Store results of queries

Provides HTTP API:

to run queries

to get result data of queries


https://github.com/tagomoris/shib

https://github.com/tagomoris/shib

Hadoop / HDFS

FluentdCluster

stream


stream

backup stream

realtime monitoring

Hive Server

Users(Web Browser)




Hadoop / HDFS

Hive Server

Users(Web Browser)

Shib (node.js)

Thrift

HTTP/Ajax

DataStore (Kyoto Tycoon)


ShibUI(non-disclosured application)

Web Front-end of Shib

Daily/Weekly/Monthly Query Management System

Graph plotting of query results

Record log to check queries no one views...

Query Builder (for hive-unfriendly engineers/directors)

(Under construction)


Hadoop / HDFS

FluentdCluster

stream


stream

backup stream

realtime monitoring

Hive Server

Users(Web Browser)




Hadoop / HDFS

Hive Server

Users(Web Browser)

Shib (node.js)

HTTP/Ajax ShibUI(Perl/Plack Web Application: Kossy)

MySQLHRForecast

HTTP


What to do nextMapReduce Job management

check query to run correctly

kill queries

Huahin Manager by @ryu_kobayashi

Hadoop MapReduce Job Manager over HTTP

http://huahin.github.com/huahin-manager/

Shib version up

node.js 0.4 based -> 0.6 based12年5月30日水曜日



Questions?


Thanks!


hive tools in nhn japan #hadoopreading

Technology