How to make your map-reduce jobs perform as well as pig: Lessons from pig optimizations
http://pig.apache.org
Thejas Nair
pig team @ Yahoo!
Apache pig PMC member
What is Pig?
Pig Latin, a high level data processing language.
An engine that executes Pig Latin locally or on a Hadoop cluster.
Pig-latin-cup pic from http://www.flickr.com/photos/frippy/2507970530/
Pig Latin example
Users = load ‘users’ as (name, age);
Fltrd = filter Users by age >= 18 and age <= 25;
Pages = load ‘pages’ as (user, url);
Jnd = join Fltrd by name, Pages by user;
Comparison with MR in Java
020406080
100120140160180
Hadoop Pig
1/20 the lines of code
0
50
100
150
200
250
300
Hadoop Pig
Minutes
1/16 the development time
What about Performance ?
Pig Compared to Map Reduce
• Faster development time• Data flow versus programming logic• Many standard data operations (e.g. join)
included• Manages all the details of connecting
jobs and data flow• Copes with Hadoop version change
issues
And, You Don’t Lose Power
• UDFs can be used to load, evaluate, aggregate, and store data
• External binaries can be invoked
• Metadata is optional
• Flexible data model
• Nested data types
• Explicit data flow programming
Pig performance
• Pigmix : pig vs mapreduce
Pig optimization principles
• vs RDBMS: There is absence of accurate models for data, operators and execution env
• Use available reliable info. Trust user choice.
• Use rules that help in most cases
• Rules based on runtime information
Logical Optimizations
Restructure given logical dataflow graph
• Apply filter, project, limit early
• Merge foreach, filter statements
• Operator rewrites
ScriptA = loadB = foreachC = filter
Logical PlanA -> B -> C
Parser Logical Optimizer
Optimized L. PlanA -> C -> B
Physical Optimizations
Physical plan: sequence of MR jobs having physical operators.
• Built-in rules. eg. use of combiner
• Specified in query - eg. join type
Optimized L. PlanX -> Y -> Z
Optimizer
Phy/MR planM(PX-PYm) R(PYr) -> M(Z)
Optimized Phy/MR Plan M(PX-PYm) C(PYc)R(PYr)->M(Z)
Translator
Hash Join
PagesPages UsersUsers
Users = load ‘users’ as (name, age);Pages = load ‘pages’ as (user, url);Jnd = join Users by name, Pages by user;
Map 1Map 1
Pagesblock nPagesblock n
Map 2Map 2
Usersblock mUsers
block m
Reducer 1Reducer 1
Reducer 2Reducer 2
(1, user)
(2, name)
(1, fred)(2, fred)(2, fred)
(1, jane)(2, jane)(2, jane)
Skew Join
PagesPages UsersUsers
Users = load ‘users’ as (name, age);Pages = load ‘pages’ as (user, url);Jnd = join Pages by user, Users by name using ‘skewed’;
Map 1Map 1
Pagesblock nPagesblock n
Map 2Map 2
Usersblock mUsers
block m
Reducer 1Reducer 1
Reducer 2Reducer 2
(1, user)
(2, name)
(1, fred, p1)(1, fred, p2)(2, fred)
(1, fred, p3)(1, fred, p4)(2, fred)
SPSP
SPSP
Merge Join
PagesPages UsersUsers
aaron . . . . . . . .zach
aaron . . . . . .zach
Users = load ‘users’ as (name, age);Pages = load ‘pages’ as (user, url);Jnd = join Pages by user, Users by name using ‘merge’;
Map 1Map 1
Map 2Map 2
UsersUsers
UsersUsers
PagesPages
PagesPages
aaron…amr
aaron…
amy…barb
amy…
Replicated Join
PagesPagesUsersUsersaaron
aaron . . . . . . .zach
aaron .zach
Users = load ‘users’ as (name, age);Pages = load ‘pages’ as (user, url);Jnd = join Pages by user, Users by name using ‘replicated’;
Map 1Map 1
Map 2Map 2
UsersUsersPagesPages
PagesPages
aaron…amr
aaron . zach
amy…barb
UsersUsersaaron . zach
Group/cogroup optimizations• On sorted and ‘collected’ data grp = group Users by name using ‘collected’;
PagesPages
aaronaaronbarneycarol . . . . . . .zach
Map 1Map 1
aaronaaronbarney
Map 2Map 2
carol . .
Multi-store scriptA = load ‘users’ as (name, age, gender, city, state);B = filter A by name is not null;C1 = group B by age, gender;D1 = foreach C1 generate group, COUNT(B);store D into ‘bydemo’;C2= group B by state;D2 = foreach C2 generate group, COUNT(B);store D2 into ‘bystate’;
A: loadA: load B: filterB: filter
C2: groupC2: group
C1: groupC1: group
C3: eval udfC3: eval udf
C2: eval udfC2: eval udf
store into ‘bystate’store into ‘bystate’
store into ‘bydemo’store into ‘bydemo’
Multi-Store Map-Reduce Planmapmap filterfilter
local rearrangelocal rearrange
splitsplit
local rearrangelocal rearrange
reducereduce
multiplexmultiplexpackagepackage packagepackage
foreachforeach foreachforeach
Memory Management
Use disk if large objects don’t fit into memory
• JVM limit > phy mem - Very poor performance
• Spill on memory threshold notification from JVM - unreliable
• pre-set limit for large bags. Custom spill logic for different bags -eg distinct bag.
Other optimizations
• Aggressive use of combiner, secondary sort
• Lazy deserialization in loaders
• Better serialization format
• Faster regex lib, compiled pattern
Future optimization work
• Improve memory management
• Join + group in single MR, if same keys used
• Even better skew handling
• Adaptive optimizations
• Automated hadoop tuning
• …
Pig - fast and flexible
More flexibility in 0.8, 0.9
• Udfs in scripting languages (python)
• MR job as relation
• Relation as scalar• Turing complete pig (0.9)
Pic courtesy http://www.flickr.com/photos/shutterbc/471935204/
Further reading
• Docs - http://pig.apache.org/docs/r0.7.0/
• Papers and talks - http://wiki.apache.org/pig/PigTalksPapers
• Training videos in vimeo.com (search ‘hadoop pig’)