BVYdde�?Xd�Midgn

gZZehl.,-2*�,/

-

-*�BVYdde�

BVYdde :) Cc�-553(�>dj �=jii c ( FjXZcZ�~ (� c�.,,,(�deZc�hdjgXZ

) Cc�.,,-(� eVX Z�HjiX ~ (�QZW�=gVlaZg�) -�XdgZ(�-A<�L G(�L C>�aZkZa�-�dc�4� YY idiVa�-N<) CcYZm c �gViZh�8�-,,�eV Zh�+�hZX) a b iZY�i Z�idiVa�cjbWZg�d �eV Zh�id�-,,�b aa dc) QZW cYZm c w) 0 w x) y Y h �­y

CcYZm c *�

N Z� hidgn�d �BVYdde .

B> M :

eVX Z�HjiX 0~hX ZbVaZhh(�YjgVWaZ(� VcYa c � V ajgZ(�VjidbVi Vaan gZWVaVcXZY

Cc�.,,/(�Add aZ�ejWa h ZY�i Z Add aZ� aZ�MnhiZb�eVeZg

Cc�.,,0(�YZkZadeZY�HjiX > hig WjiZY� aZ�MnhiZb�) >jgVWaZ�6�20G�WadX (�) BVcYa c � V ajgZ�6� z HdYZ) jidbVi XVaan�gZWVaVcXZ�6�HVbZ�HdYZ

/

B> M :

0iie6++lll*ha YZh VgZ*cZi+bVYk gjh+ Y h).443-012

B> M :

1

GVeLZYjXZ :

H> M :(

) x (�) xx () *�

!� Cc�.,,0(�uGVeLZYjXZ6�M bea ZY�>ViV�JgdXZhh c �dc�FVg Z�=ajhiZghv

2

GVe

HdYZ

LZYjXZ

GVe

GVeLZYjXZ� :

3

-*�ldgYXdjci GVeLZYjXZ *.*� (�Mea i(� (� (� (� …/*� /~ Mea i *0*�bVe � … 7 (-9 *

1*� … Zn*

2*�gZYjXZ � … kVajZ hjbcVa�kVajZ *

3*�<ZVg(.�=Vg(/�>ZZg(.�L kZg(.�cVa�7 Zn(kVajZ9� w

4RackN

NodeManager

NodeManager

NodeManager

Rack2

NodeManager

NodeManager

NodeManager

Rack1

NodeManager

NodeManager

NodeManager

C2.1

C1.4

AM2

C2.2 C2.3

AM1

C1.3

C1.2

C1.1

Hadoop Client 1

Hadoop Client 2

create app2

submit app1

submit app2

create app1

ASM Schedulerqueues

ASM Containers

NM ASM

Scheduler Resources

.......negotiates.......

.......reports to.......

.......partitions.......

ResourceManager

status report

GVeLZYjXZ� :

BVYdde� !

.,,2 BVYdde� !) B> M GVeLZYjXZ BVYdde�=dgZ�a W� �) .,,2 S BII� ­

.,,3 Nl iiZg(� VXZWdd (�F c ZYCc

.,,4 MjW�Jgd_ZXiw) BWVhZ(�Tdd ZZeZg(�J SV dd (�B kZ VXZWdd � q) =adjYZgV� Add aZ(� VXZWdd (�SV dd�~

.,,5 bVodcZ GVeLZYjXZ�

.,-- Bdgidcldg h SV dd ~

5

BVYdde**� q

GVeLZYjXZw !!!!) B> M) =ajhiZg� (�) DdW� (� egdXZhh c) =a Zci JC q!

<ViX � :) KjZgn(��AgVe �JgdXZhh c �

**� q:-,

BVYdde**� q� w !

w :) **) **� **

B> M GVeLZYjXZ :

.,-. GVeLZYjXZ�P.� S LH

--

BVYdde�.*, :

=ajhiZg� …GVeLZYjXZ� w

BVYdde

S LH

-.

BVYdde MQw :

-/

N Z�BVYdde�?XdhnhiZb�NVWaZ.,-2

-2,�~ MQiie6++ VYddeZXdhnhiZbiVWaZ* i jW* d+

BVYdde�?Xd r

-0

B> M

S LH

y

HdMKF

MKF�dc�BVYdde

x

BVYdde)gZaViZY�egd_ZXih�Vi� eVX Z

HZlMKF

x

BVYdde�?Xd (� MKF�dc�BVYdde

-1

w +

!!

:

B> M

S LH

GVeLZYjXZ

MKF�dc�BVYdde

J (�B kZ(�Mi c Zg NZo (�NV_d(�CbeVaV(�JgZhid(�BVl

Add aZ�>gZbZa.,--

>ViV�

BVYdde�?Xd (� MKF�dc�BVYdde

-2

• J � .,,4 �6� MXg ei GVeLZYjXZ�• B kZ� .,,4 �6�MKF GVeLZYjXZ�• >g aa� .,-- �6� B> M� B<VhZ(Gdc d><(M/

GVeL• CbeVaV� .,-. �6� y (� (� =adjYZgV• JgZhid� .,-/ �6� _d cw VXZWdd

• B kZ�.*,� .,-2 6�B kZ�� �NZo =C> (� ~• NV_d� .,-0 �6�>Q�dc�BVYdde(�MKF�• MeVg �MKF� .,-0 �6� y• B QK� .,-/ �6�GJJ (�

sMKF t MKF)Ic)BVYdde� : MKF -,w

:�y q

-3

B> M

GVeLZYjXZ

J � MXg ei

B kZ� MKFB kZ�.*, B kZ� �iZo

>g aa(�JgZhid

CbeVaV� y

NZo

MeVg �kh�NZo

NV_d�

BVlf

~

MeVg �MKF

q

-4y …

yNV_d(�B kZ

w:

MKF�>g aa(�CbeVaV(�JgZhid(�B QK(�

B kZ�dc�NZo

!

MeVg �L>>MeVg �MKF� L>>(�B kZ

y

q

-5

MnhiZb�HVbZVjai

NdaZgVcXZ>ncVb XMX ZYja c

Fd �N bZKjZgn

FdlFViZcXn

eVX Z�NV_d I I I I

CbeVaV R R R I

Mig c Zg�B kZ

I I I I

JgZhid R R R I

MKF)dc)BVYdde :

MKF�dc�BVYdde�

.,

MKF)Ic)BVYdde y *78�B> M� &� WVcYl Yi

-, (� 4~(�M N �3.,,�geb-, &�4~ &�-,,G<+hZX�8�4A<+hZX-*1N< B kZ(�NV_d(�CbeVaV*

hZaZXi�Xdjci & � gdb�NUN?MNVXidg " "(�"

y"(�" "� * w ww w VXidg *� B kZ -

NV_d(�CbeVaV -*1�p�/ *�-,, (�.,, w*

Q Vi�YdZh�u-,,�i bZh� VhiZg�i Vc�B kZv�VXijVaan�bZVc:

MKF)Ic)BVYdde� VXidg

.-

• BVYdde� 6� jaa�hXVc�• y 6� y gdje�Wn(�dgYZg�Wn• … 6� dgYZg�Wn kZ

gZYjXZw *

• 6�L= aZ (�IL= aZ(�eVgfjZi�• 6�IL=�

*• 6� (�• y 6�

BVYdde :

..*** ”

IL= JVgfjZi

Bdgidcldg h�

(�DVkV (�

L=� aZ

=adjYZgV�

(� (�**

(McVeen

? X Zci�>ViV�MidgV Z� dg� cVani Xh�l i � eVX Z�JVgfjZi�.*,

K �“ **

./

K �JVgfjZi�WadX �h oZw Y h WadX �h oZ� w :� ?LL�6�JVgfjZi� aZ�h djaY�cdi�WZ�hea i� cid�bjai eaZ� Y h)WadX hè B kZ eVgfjZi� (� beVaVè �JVgfjZi Y h WadX �h oZiie6++hiVX dkZg adl*Xdb+fjZhi dch+/,404331+hZi)eVgfjZi)hcVeen)djieji) aZ)h oZ) h) kZ

BVYdde�?Xd (�HdMKF :

.0HdMKF�y ~

HdMKF B> M

S LH

GVeLZYjXZ

MKF�dc�BVYdde

BVYdde ><GM q

Add aZ�< NVWaZ.,-.

B> M >< :

BWVhZ(�=VhVcYgV(�Gdc d><

HdMKF w :

.1Gdc d><(�=VhhVcYgV(�B<VhZ y

• =VhcVYgV .,,4 �6� VXZWdd �• B<VhZ .,,2 �6� ><• XXjbjad .,-- �6�B<VhZ (� ”• LZY h 6� ><�• ?aVhi XMZVgX .,-, �6�FjXZcZ• Gdc d><�6�-, Zc(� w (�Dd c� w• >ncVbd>< 6� QM (� (�MM>�

y(�

) =VX Z� LZY h ><) ZaVhi XhZVgX

HIMKF!! = J :�

.2HdMKF y !

=dch hiZcXn 6�xx w 8� w y

*

kV aVW a in 6� w 8� w

JVgi i dc�NdaZgVcXZ 6� 8�HQ

y

HdMKF J w =

= �

L><GM

=J�

w (� �B<VhZ

J�

>ncVbd(�=VhVcYgV(�=djX ><= J� = J�NZdgZb HdMKF�

HIMKF :

B> M� w

) � ­y w�

) � w

HdMKF�y ~.3

HdMKF BWVhZ

.4iie6++lll*ijidg Vahed ci*Xdb+ WVhZ+ WVhZUijidg Va*eY

BWVhZ B> M HdMKF�><

HdMKF BWVhZ

.5

HdMKF

/,

HdMKF w è u v���q�:

>< w (�F hi

HdMKF�y ~

KjZgn

HdMKF :

/-

…

-(�.*� 6� 8�-�6�-/*� 6� 8�-�6�H0*� y

L><GM… !

HdMKF

HdMKF :�

/.

EZn)PVajZ *PVajZ 0w *� ><

PVajZw Mig c (�F hi(�GVe�

LZY h(�>ncVbd><(�L V

EZn)PVajZPVajZ ~=dajbcBWVhZ(�=VhVcYgV(�>gj Y(� XXjbjad

=dajbc� Vb anPVajZ�mba(�Dhdc

Gdc d><(�=djX ><(�?aVhi XhZVgX

>dXjbZci

HdMKF

//iie6++c.b*Xd* g+c.b.,-,+YdlcadVY+=djX <VhZP-U0*eY

• Dd c Dd c w (�HdMKF… >< _d c

• HdMKF Dd c w w

HdMKF =dajbc�

/0

HdMKF >dXjbZci

/1

=dajbc�Vb an

>dXjbZci

K �=dajbc� >< :

/2>< w: Xdajbc�dg ZciZY�YWbh

=dgZw *

))

HdMKF� MKF�dc�BVYdde**� w :

/3

HdMKF Hdi�Ican�MKF … ><GM

><�

< M?�

MKFHdMKF

L><GM

MKF�dc�BVYdde

…= J� HdMKF y !

(

HdMKF� MKF�dc�BVYdde**� !

/4

B kZ MKF�FVnZg :

B> M

S LH� >ViV�IM

B< M? B kZ

MKF�=a ZciCchZgi�+�OeYViZ >ZZe�MKF� cVanh h

B<VhZ VcY�B kZ�r <ZiiZg�Nd Zi Zg

ü B< M?>ViV�

ü

ü B kZ

üKjZgn�

HdMKF�

w:

HdMKF ee *!è ‘ >< ee w *

ee�~ w * ~

P Zl�P Zl�

à w

HdMKF:

Jw =J�

/5

HIMKF!! = J :�< M? :

0,

<Vh XVaan kV aVWaZ�6� kV aVWaZ

Md i)hiViZ�6� w >ViVw Zme gZ�

?kZcijVaan�Xdch hiZcXn�6� y >ViV

= J

< M?

HdMKF y !

=dch hiZcXn�6�xx w 8� wy *

kV aVW a in�6� w 8� w*

JVgi i dc�NdaZgVcXZ 6� 8�*

HIMKF!! = J :�< M? :

0-

=C> *� (�.� … w *�

< M?� ~ *� =C>w ~(�< M? *

=C>(�< M?� = J

= J� /~ .~ *�J (� *�

‘(� ‘ Jw w =*� = :�=dch hiZcXn�

?kZcijVaan�=dch hiZci�~ *

HdMKF�y ~ r ?kZcijVaan�=dch hiZci

HZl�MKF:

0.HZlMKF :

B> M

S LH

GVeLZYjXZ

MKF�dc�BVYdde

HdMKF

HZlMKF

=C>Hdc)adX c �

h VgZY)cdi c

HdMKF VcY�L><GM …

:

Pdai><(�Hjd><(�bZbMKF(�=ajhig m

HZl�MKF :

0/

HdMKF L><GM

MKF�=C>

< o� HZlMKF ><GM(

HZlMKF y

HZlMKFLZXdgY�FdX c �ad c �<j Zg�edda�

HZl�MKF :

00

- MKF�. ��igVchVXi dch� =C>� WZ c(�gdaaWVX (�Xdbb i/ Hdc)adX c �XdcXjggZcXn�Xdcigda

NgVchVXi dc FdX cHdc)adX c w *�

&� ( HZlMKF w

0 B �eZg)cdYZ�eZg dgbVcXZ�x ><GM *

1 JVgVaaZa(�h VgZY)cdi c �VgX iZXijgZ(

w x *

HZlMKF

01

GZbdgn�

bnMKF

ü Pdai>< AHO (�GZbMKF

ü MXVaZ>< GnMKF�hXVaVW a in (ü Nd j>< AHO GVg V>< hXVaaVW a in

ü =ajhig m> Y hig WjiZY�eZZg)id)eZZg�MKF�YViVWVhZ

ü Hjd>< Y hig WjiZY�YViVWVhZ�bVcV ZbZci

s-0 2 HZlMKF=ajhig m(�bZbMKF(�Hjd><(�Mea XZ�GVX cZ(�NgVch�FVii XZ(�Pdai><

-2�HdMKF(�HZlMKF >ViVWVhZh�Nd�QViX

HZlMKF(�HdMKF(�L><GM� :

02

ü L><GM dg�cZlMKF

ü HdMKF�dg�HZlMKFü L><GM

ü HdMKF

ü < o(�Cc gV�

HZlMKF(�HdMKF(�L><GM� :

03

HZlMKF :

04

FZVYZg’ IgVXaZ(�G Xgdhd i(�M J(�C<G H X Z�JaVnZg w

…

uIcZ�h oZ�YdZh�cdi� i�Vaav

05

B> M

S LH

GVeLZYjXZ

MKF�dc�BVYdde

HdMKF

(�BVYdde�

(� �q

BVYdde

:

ajbZ(�H (� ajZciY(�ad hiVh

HZlMKF

r w :

1,

• MXg WZ .,,4 �6� VXZWdd (�• = j lV .,,4 �6�SV dd(� �• ajbZ� .,-. �6� (�y ?NF• ajZciY 6� ajbZ� eaj c� (�? E w• Fd hiVh 6 *� *� w• H .,-0 �6� ‘ HM �• MKIIJ�6�L><GM BVYdde• EV V�6�F c ZYCc(� GZhhV Z�KjZjZ > h �

< o� Fd � (� (�

) y EV Vw *? E�6�?aVhi XhZVgX � ajZciY �E WVcV Fd � gZ Vidg�

FIA

L><GMA�KjZjZ

r :

1-

w

MdjgXZ�)9�= VccZa�)9�M c) MdjgXZ�6�) = VccZa�6�) M c �6�

Jaj c

Jaj c ~w) 6�><(�JdgdidXda(�MHM(�B> M) 6�><(�KjZjZ(�

r :

1.

:

JgdidXda�

DMIH NZmi RGF

< cVgn�DMIH ­y w(�

hoZ� w(�NZmi�JVgh c �

YViV� dgbVi

N g i(� kgd(�JgdidWj ­ (�

r :

1/

egdidWj w (�i g iw *

PLI)) (�

i g i) .,,3(� VXZWdd) (� w *�

egdidWj) .,,4(�Add aZ) w (� w

N g i�kh�JgdidXda�<j Zgh�kh� kgd�) < VhZY�=dbeVg hdc

y + MigZVb c

10

B> M

S LH

GVeLZYjXZ

MKF�dc�BVYdde

HdMKF

higZVb c �

:

GZbgdn?kZci�

GL …) C+I� HQ(�> h (�

MeVg (�Midgb(�hoboV

HZlMKF

y

y r w :

11

• M0� .,-- �6�SV dd(� higZVb c � .,-/• Midgb� .,-/ �6�Nl iiZg(� higZVb c � (�• MVboV .,-0 �6�F c ZYCc(� higZVb c � (�> h �• JjahVg� .,-1 �6�ZWVn(� higZVb c � =?J(�>gj Y�

• MeVg � .,-0 �6�>ViV<g X h(� L>> � + (�GF+MKF• a c .,-0 �6�YViV gi hVch(�MeVg (�egdXZhh�

MeVg a W

y MigZVb c �< �>ViV cZmi�hiZe� dg� eVX Z� a c

a c

y w:

12

GLNZo

MeVg

a c

>ViV

>ViVMVboVMidgb

JjahVg

B> M(� aZ�HdMKF�

…MHM(�=Vaa(�

> ABVYdde�

> AL>>

13NZo

B> M� *

GVe�LZXYjXZ *

GVeLZYjXZ�kh�NZo 6�~ :

14

6�GZbdgn (�> A<ViX � �MigZVb c � w G Xgd�<ViX

GVeLZYjXZ�kh�MeVg �6�~ :

15

MigZVb c �< �>ViV6�Midgb(�MeVg �VcY�MVboV

6�GZbdgn (�> A<ViX � �MigZVb c � w G Xgd�<ViX

eVX Z�Midgb�

2,

y y

2-

y

2.

y

2/

Qdg adl(�Gdc idg c

20

BVYdde�w OC …+

MQ�!!!!

:

bWVg (�=adjYZgV�GVcV ZgIdo Z(� o VWVc(�Fj

BVYdde�

>ViV�

B> M

S LH

GVeLZYjXZ

MKF�dc�BVYdde

HdMKF

HZlMKF

y

Qdg adl�

21

OC

o VWVc

F c ZYCc�BVYdde

o VWVc

Ido Z

RGFOC

Fj

Jni dc=dYZ

Fj

Jni dc�~

Qdg adl�?c cZ�=dbeVg hdc

22

BVYdde�=ajhiZg�

=adjYZgV�GVcV Zg=adjYZgV

bWVgBdgidcQIg h

aVb c d‘ =adjY cZ

< �>ViV�

23

BVYdde�=ajhiZg� bWVg

24

BVYdde�=ajhiZg� bWVg