Hadoop Management Console
-User Account
-Share System
-Management Console
2015 中国 Hadoop 技术峰会
2015.07
About me
Gus, NieHadoop Engineer
Email: [email protected]
Oracle DBA -Teradata DBA - Hadoop Engineer –
Agenda
• Problems! Problems! Problems!
• We need Change
• HMC Solution
• Example
• Current & Next
• Q&A
Problems
dn2 dn3dn1
dn4 dn5
dn8 rank1dn7
dn2 dn3dn1
dn4 dn5
dn8 rank2dn7 dn2 dn3dn1
dn4 dn5
dn8 rank3dn7
dn2 dn3dn1
dn4 dn5
dn8 rank3dn7
dn7
dn2 dn3dn1
dn4 dn5
dn8 rank3dn7
dn7
Over 2000 nodes.
Account Increase
Problems
• Small files always there.
0
10000000
20000000
30000000
40000000
50000000
60000000
Small Files Accounts
>1GB
<1MB
<1GB
<128MB
<10MB
We need Change
• All accounts share the cluster’s resource without limit.
• Small files always there.
• Queues Usage in a mess.
• All accounts share the cluster’s resource without limit.
• Small files always there.
• Queues Access in a mess.
Resource Allocation
Self-Service
Dynamic
We need Change
Resource Allocation
Self-Service
Dynamic
Hierarchy Queue
Storage Quota
Apply Request
Auto Approve
Real time release
Pre-define capacity
We need Change
HMC Solution
HMC
Fsimage
Delimiter File
Upload to HDFS
Aggregation to
Dir ViewFilter Quota Data
Hadoop Nodes HDFS
Update Real Time Data
Parse Schedul
er
Dump Schedu
ler
Command
Console
HMC Solution
HMC
Hadoop Nodes HDFS
Remote
Scripts
Set Name Quota
Set Space Quota
Dump to Schedule
r
Scheduler
XMLRefresh
Scheduler
Command
Console
• Small files always there.
Current & Next
0
2000000
4000000
6000000
8000000
10000000
12000000
14000000
16000000
18000000
Small Files Accounts
>1GB
<1MB
<1GB
<128MB
<10MB
• Time Window Scheduler.
• Dynamic mapping overwrite.
• Account full function self-service.
• ACL for HDFS
Current & Next