dynamic allocation in spark

39
'\QDPLF $OORFDWLRQ LQ (ODVWLF DOORFDWLRQ RI FOXVWHU UHVRXUFHV WR \RXU DSSOLFDWLRQ +!/"4 / 2+" ǖǖ1%Ǿ ǗǕǖǚ

Upload: databricks

Post on 28-Jul-2015

930 views

Category:

Software


1 download

TRANSCRIPT

Page 1: Dynamic Allocation in Spark

'\QDPLF�$OORFDWLRQ�LQ(ODVWLF�DOORFDWLRQ�RI�FOXVWHU�UHVRXUFHV�WR�\RXU�DSSOLFDWLRQ

�+!/"4��/ 2+"�ǖǖ1%Ǿ�ǗǕǖǚ

Page 2: Dynamic Allocation in Spark

�,4�#�*&)&�/��/"�6,2�4&1%��-�/(Ȅ

�ǽ �� ,+1/&�21"�1,�&1

�ǽ ��20"�&1�&+�-/,!2 1&,+

ǽ ���*�"3�)2�1&+$�&1

!ǽ ��%�3"�+,1%&+$�1,�!,�4&1%�&1

Ǘ

Page 3: Dynamic Allocation in Spark

�%�1�&0����������������Ȅ

ǘ

�01��+!�$"+"/�)�"+$&+"�#,/��&$�!�1��-/, "00&+$

�01�1,�/2+� ,!"± �+Ȓ*"*,/6�!�1��0%�/&+$± "+"/�)� ,*-21�1&,+�$/�-%0

�01�1,�4/&1"� ,!"± �& %����0�&+� �3�Ǿ�� �)�Ǿ��61%,+± �+1"/� 1&3"�0%"))

Page 4: Dynamic Allocation in Spark

Ǚ

�-�/(��,/"

�-�/(��1/"�*&+$

/"�)Ȓ1&*"

�-�/(����01/2 12/"!�!�1�

��)&�*� %&+"�)"�/+&+$

/�-%�$/�-%

«

�%�1�&0����������������Ȅ

Page 5: Dynamic Allocation in Spark

��,21���1��/& (0

ǚ

,2+!"!��6� /"�1,/0�,#��-�/(��+!�/"*�&+0�)�/$"01� ,+1/&�21,/

�##"/0���%,01"!�0"/3& "± �-�/(�,+���Ǘ± �,1"�,,(0Ȕ �),1�3&02�)&7�1&,+0± �)201"/�*�+�$"*"+1± � %"!2)"!�',�0

Page 6: Dynamic Allocation in Spark

�+�1%&0�1�)(

Ǜ

�-�/(�"5" 21&,+�,3"/3&"4

�6+�*& ��)), �1&,+

�"*,

�+!201/6�20�$"

Page 7: Dynamic Allocation in Spark

&OXVWHU

&OXVWHU�0DQDJHU

&38���0HP

����

6SDUN�RYHUYLHZ

'ULYHU

ǜ

Page 8: Dynamic Allocation in Spark

ǝ

'ULYHU

&OXVWHU

&OXVWHU�0DQDJHU

&38���0HP

����

6SDUN�RYHUYLHZ

YDO�VF� �QHZ�6SDUN&RQWH[W�����VF�WH[W)LOH��ZRUGV�W[W�����IODW0DS�^�OLQH� !�OLQH�VSOLW������`���PDS�^�ZRUG� !��ZRUG�����`���UHGXFH%\.H\�^�FDVH��D��E�� !�D���E�`���VDYH$V7H[W)LOH��RXWSXW�W[W�����

;SVH'SYRX�WGEPE

Page 9: Dynamic Allocation in Spark

&OXVWHU

&OXVWHU�0DQDJHU

&38���0HP

����

6SDUN�RYHUYLHZ

'ULYHU

([HFXWRU

([HFXWRU:RUG&RXQW�VFDOD

Ǟ

Page 10: Dynamic Allocation in Spark

&OXVWHU

&OXVWHU�0DQDJHU

&38���0HP

����

6SDUN�RYHUYLHZ

([HFXWRU

([HFXWRU:RUG&RXQW�VFDOD

ǖǕ

'ULYHU

,��ǖ

Page 11: Dynamic Allocation in Spark

&OXVWHU

&OXVWHU�0DQDJHU

&38���0HP

����

6SDUN�RYHUYLHZ

([HFXWRU

([HFXWRU:RUG&RXQW�VFDOD

ǖǖ

'ULYHU([HFXWRUV�UHPDLQ�DIWHU�-RE���ILQLVKHV�DQG�DUH�ODWHU�UHXVHG�IRU�IXWXUH�MREV�

7KLV�DOORZV�IXWXUH�MREV�WR�DFFHVV�LQWHUPHGLDWH�GDWD�FDFKHG�LQ�PHPRU\�

Page 12: Dynamic Allocation in Spark

&OXVWHU

&OXVWHU�0DQDJHU

&38���0HP

����

6SDUN�RYHUYLHZ

([HFXWRU

([HFXWRU:RUG&RXQW�VFDOD

ǖǗ

'ULYHU

,��Ǘ

Page 13: Dynamic Allocation in Spark

&OXVWHU

&OXVWHU�0DQDJHU

&38���0HP

����

6SDUN�RYHUYLHZ

([HFXWRU

([HFXWRU:RUG&RXQW�VFDOD

ǖǘ

'ULYHU

'ULYHU

Page 14: Dynamic Allocation in Spark

'ULYHU

ǖǙ

&OXVWHU

&OXVWHU�0DQDJHU

'49���1IQ

���

6SDUN�RYHUYLHZ

'ULYHU

([HFXWRU

([HFXWRU:RUG&RXQW�VFDOD

:HOFRPH�WR������BBBB��������������BB�������BB�BB��BBB�BBBBB���BB����B?�?��B�?��B�C��BB���B�����BBB���BB�?B�B�B���B�?B?���YHUVLRQ�������������B�

8VLQJ�6FDOD�YHUVLRQ���������-DYD�+RW6SRW�70�����%LW�6HUYHU�90��-DYD������B���6SDUN�FRQWH[W�DYDLODEOH�DV�VF�

VFDOD!�YDO�UGG� �VF�WH[W)LOH��KGIV���QDPHV�MVRQ�����7KLV�LV�D�ORQJ�UXQQLQJ�DSS

VSDUN�VKHOO

Page 15: Dynamic Allocation in Spark

&OXVWHU

&OXVWHU�0DQDJHU

'49���1IQ

����

6SDUN�RYHUYLHZ

'ULYHU

([HFXWRU

([HFXWRU:RUG&RXQW�VFDOD

VSDUN�VKHOO

'ULYHU

Page 16: Dynamic Allocation in Spark

&OXVWHU

&OXVWHU�0DQDJHU

&38���0HP

����

6SDUN�RYHUYLHZ

'ULYHU

([HFXWRU:RUG&RXQW�VFDOD

([HFXWRU

([HFXWRU([HFXWRU

([HFXWRU

VSDUN�VKHOO

'ULYHU

Page 17: Dynamic Allocation in Spark

&OXVWHU

&OXVWHU�0DQDJHU

&38���0HP

����

6SDUN�RYHUYLHZ

'ULYHU

([HFXWRU:RUG&RXQW�VFDOD

([HFXWRU

([HFXWRU([HFXWRU

([HFXWRU

VSDUN�VKHOO

'ULYHU

VSDUN�VKHOO�EHFRPHV�LGOH�EXW�FRQWLQXHV�WR�KROG�RQWR�DOO�FOXVWHU�UHVRXUFHV�

0HDQZKLOH��RWKHU�DSSOLFDWLRQV�FDQQRW�EH�ODXQFKHG�RQ�WKH�FOXVWHU�

Page 18: Dynamic Allocation in Spark

6SDUN�UHXVHV�FOXVWHU�UHVRXUFHV

(IILFLHQW�GDWD�VKDULQJ)LQH�JUDLQHG�SDUWLWLRQLQJ/RZ�ODWHQF\�VFKHGXOLQJ

«�EXW�PD\�KROG�RQWR�UHVRXUFHV�IRU�D�ORQJ�WLPH�

Page 19: Dynamic Allocation in Spark

5HVRXUFH�&38���0HP�

7LPH

$OORFDWHG8VHG

6WDWLF�DOORFDWLRQ

0RUH�UHVRXUFHV�DOORFDWHG�WKDQ�XVHG

Page 20: Dynamic Allocation in Spark

5HVRXUFH�&38���0HP�

7LPH

$OORFDWHG8VHG

'\QDPLF�DOORFDWLRQ

0RUH�HIILFLHQW�XWLOL]DWLRQ�RI�FOXVWHU�UHVRXUFHV

Page 21: Dynamic Allocation in Spark

:K\�LV�WKLV�D�6SDUN�SUREOHP"

([HFXWLRQ�PRGHO�LV�GLIIHUHQW�IURP�V\VWHPV�OLNH�05

Page 22: Dynamic Allocation in Spark

:K\�LV�WKLV�D�6SDUN�SUREOHP"

([HFXWLRQ�PRGHO�LV�GLIIHUHQW�IURP�V\VWHPV�OLNH�05

6SDUN�UHXVHV�ORQJ�UXQQLQJ�H[HFXWRUV�IRU�VSHHG

05�ODXQFKHV�VKRUW�OLYHG�FRQWDLQHUV�IRU�HDFK�WDVN

Page 23: Dynamic Allocation in Spark

'\QDPLF�DOORFDWLRQ

«�DOORZV�\RXU�6SDUN�DSSOLFDWLRQ�WR�VFDOH�WKH�QXPEHU�RI�H[HFXWRUV�XS�DQG�GRZQ�EDVHG�RQ�WKH�ZRUNORDG

,I�H[HFXWRUV�DUH�LGOH��UHPRYH�WKHP�,I�ZH�QHHG�PRUH�H[HFXWRUV��UHTXHVW�WKHP�

Page 24: Dynamic Allocation in Spark

6FDOH�XS�SROLF\

5HTXHVW�H[HFXWRUV�ZKHQ�WKHUH�DUH�SHQGLQJ�WDVNV

6FDOH�XS�LQ�PXOWLSOH�URXQGV

([SRQHQWLDO�LQFUHDVH�LQ���H[HFXWRUV�RYHU�WLPH

��H[HFXWRUV

7LPH

Page 25: Dynamic Allocation in Spark

6FDOH�XS�SROLF\

([SRQHQWLDO�LQFUHDVH�ZKLOH�WKHUH�DUH�SHQGLQJ�WDVNV

:K\"

Ɣ �6ORZ�VWDUW��LQ�FDVH�ZH�RQO\�QHHG�D�IHZ�PRUH�H[HFXWRUV

Ɣ ([SRQHQWLDO�JURZWK�LQ�FDVH�LW�WXUQV�RXW�WKDW�ZH�QHHG�PDQ\

Page 26: Dynamic Allocation in Spark

6FDOH�GRZQ�SROLF\

5HPRYH�H[HFXWRUV�WKDW�KDYH�EHHQ�LGOH�IRU�1�VHFRQGV��H[HFXWRUV

7LPH

LGOH

LGOH

Page 27: Dynamic Allocation in Spark

'\QDPLF�DOORFDWLRQ�H[DPSOH��H[HFXWRUV

7LPH

Page 28: Dynamic Allocation in Spark

'\QDPLF�DOORFDWLRQ�XVH�FDVHV

/RQJ�UXQQLQJ�(7/�MREV(�J��3DUVLQJ�3%�RI�-621�LQWR�3DUTXHW�LQ�6�

,QWHUDFWLYH�DSSOLFDWLRQV���VHUYHU(�J��6SDUN�VKHOO��2R\DOD�MRE�VHUYHU

$Q\�DSSOLFDWLRQ�ZLWK�ODUJH�VKXIIOHV

Page 29: Dynamic Allocation in Spark

'\QDPLF�DOORFDWLRQ�VHWXS

VSDUN�G\QDPLF$OORFDWLRQ�HQDEOHGVSDUN�VKXIIOH�VHUYLFH�HQDEOHG

��2SWLRQDO��VSDUN�G\QDPLF$OORFDWLRQ�PLQ([HFXWRUVVSDUN�G\QDPLF$OORFDWLRQ�PD[([HFXWRUVVSDUN�G\QDPLF$OORFDWLRQ�LQLWLDO([HFXWRUVVSDUN�G\QDPLF$OORFDWLRQ�H[HFXWRU,GOH7LPHRXWVSDUN�G\QDPLF$OORFDWLRQ�VFKHGXOHU%DFNORJ7LPHRXWVSDUN�G\QDPLF$OORFDWLRQ�VXVWDLQHG6FKHGXOHU%DFNORJ7LPHRXW

WUXHWUXH

����V�V�V

Page 30: Dynamic Allocation in Spark

'\QDPLF�DOORFDWLRQ�VHWXS

$GGLWLRQDOO\��VHW�XS�H[WHUQDO�VKXIIOH�VHUYLFH

)RU�PRUH�GHWDLO�KWWS���VSDUN�DSDFKH�RUJ�GRFV�ODWHVW�MRE�VFKHGXOLQJ�KWPO�G\QDPLF�UHVRXUFH�DOORFDWLRQ

Page 31: Dynamic Allocation in Spark

'HPR

Page 32: Dynamic Allocation in Spark
Page 33: Dynamic Allocation in Spark

&DVH�VWXG\����7%�PHPRU\�ZLWK�������QRGHV

����3%�6��ZDUHKRXVH����3%�3DUTXHW�

'\QDPLF�DOORFDWLRQ�ZLWK�XS�WR��������H[HFXWRUV

(QDEOH�G\QDPLF�DOORFDWLRQ�IRU�DOO�6SDUN�DSSOLFDWLRQV

5XQ�6SDUN�DORQJVLGH�+LYH��3LJ��0DS5HGXFH

8VHG�IRU�DG�KRF�TXHU\�DQG�H[SHULPHQWDWLRQ

�WKLV�WHDP�RQO\

Page 34: Dynamic Allocation in Spark

&DVH�VWXG\

�����7%�PHPRU\�ZLWK�������QRGHV

����3%�GDWD�ZDUHKRXVH

8VH�G\QDPLF�DOORFDWLRQ�ZLWK�XS�WR������H[HFXWRUV

3ULPDU\�XVH�FDVHV�DUH�(7/�DQG�64/

5XQ�6SDUN�DORQJVLGH�6WRUP��0DS5HGXFH��3LJ

Page 35: Dynamic Allocation in Spark

'\QDPLF�DOORFDWLRQ�VWDWXV

$YDLODEOH�VLQFH�6SDUN������

.QRZQ�LVVXH�LQ�6SDUN��������IL[HG�LQ������

6WDEOH�LQ�6SDUN�������RQZDUGV

&DFKLQJ�VXSSRUW�FRPLQJ�LQ������

Page 36: Dynamic Allocation in Spark

0DQXDO�G\QDPLF�DOORFDWLRQ

���5HTXHVW���H[WUD�H[HFXWRUVVF�UHTXHVW([HFXWRUV���

���.LOO�H[HFXWRUV�ZLWK�,'V��������DQG���VF�NLOO([HFXWRUV�6HT������������������

1HZ�GHYHORSHU�$3,�DV�RI�6SDUN����

Page 37: Dynamic Allocation in Spark

)XWXUH

6XSSRUW�IRU�0HVRV�PRGH�63$5.�������35�UHDG\�

6XSSRUW�IRU�VWDQGDORQH�PRGH�63$5.�������35�VRRQ�

%HWWHU�VXSSRUW�IRU�FDFKLQJ�63$5.�������35�PHUJHG��

,QWHJUDWLRQ�ZLWK�FORXG�FRPSXWH�SODWIRUP

Page 38: Dynamic Allocation in Spark

6XPPDU\

6SDUN�LV�IDVW�EHFDXVH�LW�UHXVHV�FOXVWHU�UHVRXUFHV

6SDUN�KROGV�RQWR�UHVRXUFHV�IRU�D�ORQJ�WLPH

'\QDPLF�DOORFDWLRQ�DOORZV�\RX�WR�EHQHILW�IURP�WKH�IRUPHU�ZLWKRXW�VXIIHULQJ�WKH�ODWWHU

Page 39: Dynamic Allocation in Spark

7KDQN�\RX�+!/"4ȯ!�1��/& (0ǽ ,*