hadoop 130419075715-phpapp02(1)
TRANSCRIPT
![Page 1: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/1.jpg)
HadoopEdit by Cassell Hsu
2013.04.19
![Page 2: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/2.jpg)
Hadoop
Master
Slave
Slave
Slave
![Page 3: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/3.jpg)
Hadoop
•Master
•NameNode
•JobTracker
•SecondaryNameNode
![Page 4: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/4.jpg)
NameNode•Where is NameNode?
•Master
•HDFS(Hadoop Distributed File System)
•What is NameNode?
•資料之位置資訊
•資料之屬性
??
![Page 5: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/5.jpg)
NameNode
•位置資訊 ?
•所有資料皆存放在 – DataNode
![Page 6: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/6.jpg)
DataNode
•What is DataNode
•存放資料•Where is DataNode
•HDFS
•Slaves (and Master)
![Page 7: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/7.jpg)
UserUser
DataNodeDataNode
NameNodeNameNode
DataNodeDataNode
DataNodeDataNode
128Mb
B64Mb
A64Mb
Check hdfs-site.xml
B64Mb
A64Mb
![Page 8: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/8.jpg)
Hadoop
DataNodeDataNodeNameNodeNameNode
MasterDataNodeDataNode
DataNodeDataNode
DataNodeDataNode
Slaves
![Page 9: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/9.jpg)
Hadoop
•Master
•NameNode
•JobTracker
•SecondaryNameNode
![Page 10: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/10.jpg)
JobTracker
•What is JobTracker?
•排程工作•Where is JobTracker?
•Master
誰來工作?
![Page 11: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/11.jpg)
JobTracker & TaskTrackerJobTracker TaskTracker
Where Master Slaves
What 排程工作 執行工作
![Page 12: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/12.jpg)
Hadoop
DataNodeDataNodeNameNodeNameNode
MasterDataNodeDataNode
Slaves
JobTrackerJobTracker
TaskTrackerTaskTracker
![Page 13: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/13.jpg)
Hadoop
•Master
•NameNode
•JobTracker
•SecondaryNameNode
![Page 14: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/14.jpg)
SecondaryNameNode
•What is SecondaryNameNode?
•NameNode發生錯誤時補救
•Where is SecondaryNameNode?
•Master
![Page 15: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/15.jpg)
Hadoop
DataNodeDataNodeNameNodeNameNode
MasterDataNodeDataNode
Slaves
JobTrackerJobTracker
TaskTrackerTaskTracker
SecondaryNodeSecondaryNode
![Page 16: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/16.jpg)
MapReduce
User Master
Slave2
Slave1
A
A1
A2Task
NameNode
Task
Result1
Result2
Reduce
Final Result
HDFS
![Page 17: Hadoop 130419075715-phpapp02(1)](https://reader035.vdocuments.pub/reader035/viewer/2022062712/55d4bf7ebb61eb52338b4581/html5/thumbnails/17.jpg)
MapReduce
•檔案切割•Hadoop上區塊切割
•程式指定