map reduce
TRANSCRIPT
The Map Reduce Programming Model
Boris Farber
04/12/2023
04/12/2023
About Me
• Senior Software Engineer at Varonis– Data Governance, Leakage and
Synchronization– Android , Windows
• Author of Profiterole – Map Reduce library for Android
• “Android System Programming” book reviewer
Credits
• Avner Ben– Chief Architect at Elisra Systems– Skill Tree ® designer
• Nathan Marz– Author of “Big Data” book at Manning– Software Engineer at Twitter– Creator of Storm and Cascalog
• Matei Zaharia – – http://www.cs.berkeley.edu/~matei/
Plan
• Introduction to Big Data - problem• Map Reduce - solution
– Map/Reduce functions– Sample problems– Analysis– Cascalog
• Lambda Architecture - getting the big picture
• Summary
What is Big Data
• Big Data is a new programming paradigm to support data flow programming where the traditional RDBMS and SQL based systems fail, not only to scale up but also to provide desired functionality.
• For example back bone of Facebook/Twitter/LinkedIn/Google
• The common data pattern for the companies above huge amount of un-structured data such as tweets, likes, network updates.
Big Data
• The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures.
• To gain value from this data, you must choose an alternative way to process it.
• Processing the data to keep pre-computed queries (we don’t want to perform peta-byte search for every query)
Handling Huge Data Amounts
• Cost-efficiency:– Commodity machines (cheap, but unreliable)– Commodity network– Automatic fault-tolerance (fewer
administrators)– Easy to use (fewer programmers)
• Large-Scale Data Processing with commodity hardware
• Want to use 1000s of CPUs/Processes/Threads• But don’t want hassle of managing things
www.recessframework.org/page/map-reduce-anonymous-functions-lambdas-php
Map Reduce
• A programming model (& its associated implementation) for processing large data set.
• Exploits large set of commodity computers• Executes process in distributed manner• Offers high degree of transparencies• Based on Functional Programming
approach which is inherently more scalable than Object Oriented one !
Challenge
• The reason why MapReduce is such a powerful paradigm is because programs written in terms of MapReduce are inherently scalable.
• The same program can run on ten megabytes of data as can run on ten petabytes of data, why ?
Usage
• http://research.google.com/archive/mapreduce.html Original paper by Google
• Backbone of Hadoop – open source Apache Big Data framework
Map
Original list
Function
New List
Map
• Is a higher-order function that applies a given function to each element of a list, returning a list of results. It is often called apply-to-all when considered in functional form.
• (defn bubble[x] (* x x))• (map #(bubble %1) [ 1 3 5 7 ])
(1 9 25 49)
Reduce
Original list
Function
Result1000
Reduce
• Reduce and accumulate are a family of higher-order functions that analyze a data structure and recombine through use of a given combining operation the results of recursively processing its constituent parts, building up a return value.
• (reduce * [1 2 3 4 5 6 6]) 4320
Map-Reduce
1."Map" step: The master node takes the input, partitions it up into smaller sub-problems, and distributes them to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker node processes the smaller problem, and passes the answer back to its master node.
2."Reduce" step: The master node then collects the answers to all the sub-problems and combines them in some way to form the output – the answer to the problem it was originally trying to solve.
Word Count Naïve Approach
• You can do this sequentially by getting your fastest machine (you've got plenty lying around) and running over the text from start to finish.
• Maintain a hash map of every word you find (the key) and incrementing the frequency (value) every time you parse a word.
• Simple, straightforward and slow.
Word Count Execution
the quick
brown fox
the fox ate
the mouse
how now
brown cow
MapMap
MapMap
MapMap
ReduceReduce
ReduceReduce
brown, 2
fox, 2
how, 1
now, 1
the, 3
ate, 1
cow, 1
mouse, 1
quick, 1
the, 1brown, 1
fox, 1
quick, 1
the, 1fox, 1the, 1
how, 1now, 1
brown, 1
ate, 1mouse, 1
cow, 1
InputMap Reduce
Reduce Output
http://code.google.com/p/lokad-cloud/wiki/MapReduceSample
Search• Input: (lineNumber, line) records• Output: lines matching a given pattern
• Map: if(line matches pattern)
output(line)• Reduce: identify function
– Alternative: no reducer (map-only job)
Inverted Index• Input: (filename, text) records• Output: list of files containing each word
• Map: foreach word in text.split() output(word, filename)
• Combine: uniquify filenames for each word
• Reduce:def reduce(word, filenames) output(word, sort(filenames))
Inverted Index Example
to be or not to be afraid, (12th.txt)
be, (12th.txt, hamlet.txt)greatness, (12th.txt)
not, (12th.txt, hamlet.txt)of, (12th.txt)
or, (hamlet.txt)to, (hamlet.txt)
hamlet.txt
be not afraid of greatness
12th.txt
to, hamlet.txtbe, hamlet.txtor, hamlet.txtnot, hamlet.txt
be, 12th.txtnot, 12th.txtafraid, 12th.txtof, 12th.txtgreatness, 12th.txt
Map Reduce
• Map:– Accepts input
key/value pair– Emits intermediate
key/value pair
• Reduce :– Accepts intermediate
key/value* pair– Emits output
key/value pair
Very big
data
ResultMAP
REDUCE
PartitioningFunction
Programming Model
• Data type: key-value records
• Map function:
(Kin, Vin) list(Kinter, Vinter)
• Reduce function:
(Kinter, list(Vinter)) list(Kout, Vout)
Throwing Hardware Linear Scalability
• Cheap nodes fail, especially if you have many
• Mean time between failures for 1 node = 3 years
• Mean time between failures for 1000 nodes = 1 day
• Programming distributed systems is hard users write “map” & “reduce” functions, system distributes work and handles faults
Fault Tolerance Task Crash
• If a task crashes:– Retry on another node
• OK for a map because it has no dependencies• OK for reduce because map outputs are on disk
– If the same task fails repeatedly, fail the job or ignore that input block (user-controlled)
• For these fault tolerance features to work, your map and reduce tasks must be side-effect-free
Fault Tolerance Node Crash
• If a node crashes:– Re-launch its current tasks on other nodes– Re-run any maps the node previously ran, Necessary
because their output files were lost along with the crashed node
Fault Tolerance Slow Task
• If a task is going slowly (straggler):– Launch second copy of task on another node (“speculative
execution”)– Take the output of whichever copy finishes first, and kill the
other
• Surprisingly important in large clusters– Stragglers occur frequently due to failing hardware,
software bugs, misconfiguration, etc– Single straggler may noticeably slow down a job
Key Code (async thread pool)MapCallback<TMapInput> mapper = new MapCallback<TMapInput>();List<HashMap<String, Integer>> maps = new LinkedList<HashMap<String,
Integer>>();int numThreads = 25;ExecutorService pool = Executors.newFixedThreadPool(numThreads);CompletionService<OutputUnit> futurePool = new
ExecutorCompletionService<MapCallback.OutputUnit>(pool);Set<Future<OutputUnit>> futureSet = new HashSet<Future<OutputUnit>>();
// linear addition of jobs, parallel executionfor (TMapInput m : input) {
futureSet.add(futurePool.submit(mapper.makeWorker(m)));}// tasks runningpool.shutdown();
Clojure/Cascalog
• Key problem in Java – very low level processing with lack of high level combinations (it took me while to add another reducer …) static typing and OO (function is not first class citizen) felt writing interpreter
• Is a fully-featured data processing and querying library for Clojure or Java.
• The main use cases for Cascalog are processing "Big Data" on top of Hadoop or doing analysis on your local computer.
• Cascalog is a replacement for tools like Pig, Hive, and Cascading and operates at a significantly higher level of abstraction than those tools.
Benefits of Map Reduce
• Many problems can be phrased this way• Elegant and Powerful
Takeaways
• By providing a data-parallel programming model, MapReduce can control job execution in useful ways:– Automatic division of job into tasks– Automatic load balancing– Recovery from failures & stragglers
• User focuses on application, not on complexities of distributed computing
Conclusions
• MapReduce programming model hides the complexity of work distribution and fault tolerance
• Principal design philosophies:– Make it scalable, so you can throw hardware at
problems– Make it cheap, lowering hardware, programming and
admin costs
• MapReduce is not suitable for all problems, but when it works, it may save you quite a bit of time
Suitable for your task if
• Have a cluster• Working with large dataset• Working with independent data (or
assumed)• Can be cast into map and reduce
Lambda Architecture
Lambda Architecture
• Query = Function(All Data)• General purpose approach to implementing an
arbitrary function on an arbitrary large dataset and having the function return its results with low latency
• The Lambda Architecture solves the problem of computing arbitrary functions on arbitrary data in real-time by decomposing the problem into three layers: the batch layer, the serving layer, and the speed layer.
• Key Insight The alternative approach is to pre-compute the query function – Why ?
Chart
Lambda Architecture
• Batch Layer – pre- computed saved queries (saves time), using Map Reduce
• Serving layer – updates• Speed layer – combined result (both data
in database plus updates not yet inserted)
THANK YOU!