mapreduce 資工碩一 黃威凱. outline purpose example method advanced 資工碩一 黃威凱

資工碩一 黃威凱

MapReduce

資工碩一 黃威凱

OutlinePurposeExampleMethodAdvanced

資工碩一 黃威凱

PURPOSE

資工碩一 黃威凱

PurposeData miningData processing

資工碩一 黃威凱

EXAMPLE

資工碩一 黃威凱

ExampleFind the maximum temperature of

yearNational Climatic Data Center(NCDC)

◦The data is stored using a line-oriented ASCII format , in which each line is a record

◦There is a directory for each year from 1901 to 2001 ,each containing a gzipped file for each weather station with its readings for that year

資工碩一 黃威凱

Example(Data format)

資工碩一 黃威凱

Example(Gzipped file, example for 1990)

◦% ls raw/1990 | head◦010010-99999-1990.gz◦010014-99999-1990.gz◦010015-99999-1990.gz◦010016-99999-1990.gz◦010017-99999-1990.gz◦010030-99999-1990.gz◦010040-99999-1990.gz◦010080-99999-1990.gz◦010100-99999-1990.gz◦010150-99999-1990.gz

資工碩一 黃威凱

METHOD

資工碩一 黃威凱

MethodAnalzing the data with Unix toolsAnalzing the data with Hadoop

資工碩一 黃威凱

Method(Unix tools)

資工碩一 黃威凱

Method(Unix tools)Here is the beginning of a run:

◦% ./max_temperature.sh◦1901 317◦1902 244◦1903 289◦1904 256◦1905 283◦ ...

The complete run for the century took 42 minutes in one run single EC2 High-CPU Extra Large Instance.

資工碩一 黃威凱

Method(Hadoop)Use MapReduce

◦Map Shuffle

◦Reduce

資工碩一 黃威凱

Method(Hadoop)Map function

◦Pull out the year and the air temperature

◦Transform key-value pairs

資工碩一 黃威凱

Method(Hadoop)Map function

◦The shuffle Each reduce task is fed by many map

tasks.

資工碩一 黃威凱

Method(Hadoop)Reduce function

◦Iterate through the list and pick up the maximum reading

◦Input (1949, [111, 78]) (1950, [0, 22, -11])

◦Output: (1949, 111) (1950, 22)

資工碩一 黃威凱

Method(Hadoop)Data flow

資工碩一 黃威凱

Method(Hadoop)Java MapReduce-Mapper

example

資工碩一 黃威凱

Method(Hadoop)Java MapReduce-Reduce example

資工碩一 黃威凱

Method(Hadoop)Java MapReduce-Job example

Support multiple path

資工碩一 黃威凱

ADVANCED

資工碩一 黃威凱

AdvancedCase1

資工碩一 黃威凱

AdvancedCase2

資工碩一 黃威凱

AdvancedCase3

資工碩一 黃威凱

AdvancedCombiner Functions on Map

output◦Example

Map input1: (1950, 0), (1950, 20), (1950, 10)

Map input2: (1950, 25), (1950, 15) After shuffle:

Map1: (1950, [0,20,10]) Map2: (1950, [25,15])

No Use Combiner to reduce input (1950, [0, 20, 10, 25, 15])

Use Combiner to reduce input (1950, [20, 25])