dynamic allocation in spark
TRANSCRIPT
'\QDPLF�$OORFDWLRQ�LQ(ODVWLF�DOORFDWLRQ�RI�FOXVWHU�UHVRXUFHV�WR�\RXU�DSSOLFDWLRQ
�+!/"4��/ 2+"�ǖǖ1%Ǿ�ǗǕǖǚ
�,4�#�*&)&�/��/"�6,2�4&1%��-�/(Ȅ
�ǽ �� ,+1/&�21"�1,�&1
�ǽ ��20"�&1�&+�-/,!2 1&,+
ǽ ���*�"3�)2�1&+$�&1
!ǽ ��%�3"�+,1%&+$�1,�!,�4&1%�&1
Ǘ
�%�1�&0����������������Ȅ
ǘ
�01��+!�$"+"/�)�"+$&+"�#,/��&$�!�1��-/, "00&+$
�01�1,�/2+� ,!"± �+Ȓ*"*,/6�!�1��0%�/&+$± "+"/�)� ,*-21�1&,+�$/�-%0
�01�1,�4/&1"� ,!"± �& %����0�&+� �3�Ǿ�� �)�Ǿ��61%,+± �+1"/� 1&3"�0%"))
Ǚ
�-�/(��,/"
�-�/(��1/"�*&+$
/"�)Ȓ1&*"
�-�/(����01/2 12/"!�!�1�
��)&�*� %&+"�)"�/+&+$
/�-%�$/�-%
«
�%�1�&0����������������Ȅ
��,21���1��/& (0
ǚ
,2+!"!��6� /"�1,/0�,#��-�/(��+!�/"*�&+0�)�/$"01� ,+1/&�21,/
�##"/0���%,01"!�0"/3& "± �-�/(�,+���Ǘ± �,1"�,,(0Ȕ �),1�3&02�)&7�1&,+0± �)201"/�*�+�$"*"+1± � %"!2)"!�',�0
�+�1%&0�1�)(
Ǜ
�-�/(�"5" 21&,+�,3"/3&"4
�6+�*& ��)), �1&,+
�"*,
�+!201/6�20�$"
&OXVWHU
&OXVWHU�0DQDJHU
&38���0HP
����
6SDUN�RYHUYLHZ
'ULYHU
ǜ
ǝ
'ULYHU
&OXVWHU
&OXVWHU�0DQDJHU
&38���0HP
����
6SDUN�RYHUYLHZ
YDO�VF� �QHZ�6SDUN&RQWH[W�����VF�WH[W)LOH��ZRUGV�W[W�����IODW0DS�^�OLQH� !�OLQH�VSOLW������`���PDS�^�ZRUG� !��ZRUG�����`���UHGXFH%\.H\�^�FDVH��D��E�� !�D���E�`���VDYH$V7H[W)LOH��RXWSXW�W[W�����
;SVH'SYRX�WGEPE
&OXVWHU
&OXVWHU�0DQDJHU
&38���0HP
����
6SDUN�RYHUYLHZ
'ULYHU
([HFXWRU
([HFXWRU:RUG&RXQW�VFDOD
Ǟ
&OXVWHU
&OXVWHU�0DQDJHU
&38���0HP
����
6SDUN�RYHUYLHZ
([HFXWRU
([HFXWRU:RUG&RXQW�VFDOD
ǖǕ
'ULYHU
,��ǖ
&OXVWHU
&OXVWHU�0DQDJHU
&38���0HP
����
6SDUN�RYHUYLHZ
([HFXWRU
([HFXWRU:RUG&RXQW�VFDOD
ǖǖ
'ULYHU([HFXWRUV�UHPDLQ�DIWHU�-RE���ILQLVKHV�DQG�DUH�ODWHU�UHXVHG�IRU�IXWXUH�MREV�
7KLV�DOORZV�IXWXUH�MREV�WR�DFFHVV�LQWHUPHGLDWH�GDWD�FDFKHG�LQ�PHPRU\�
&OXVWHU
&OXVWHU�0DQDJHU
&38���0HP
����
6SDUN�RYHUYLHZ
([HFXWRU
([HFXWRU:RUG&RXQW�VFDOD
ǖǗ
'ULYHU
,��Ǘ
&OXVWHU
&OXVWHU�0DQDJHU
&38���0HP
����
6SDUN�RYHUYLHZ
([HFXWRU
([HFXWRU:RUG&RXQW�VFDOD
ǖǘ
'ULYHU
'ULYHU
'ULYHU
ǖǙ
&OXVWHU
&OXVWHU�0DQDJHU
'49���1IQ
���
6SDUN�RYHUYLHZ
'ULYHU
([HFXWRU
([HFXWRU:RUG&RXQW�VFDOD
:HOFRPH�WR������BBBB��������������BB�������BB�BB��BBB�BBBBB���BB����B?�?��B�?��B�C��BB���B�����BBB���BB�?B�B�B���B�?B?���YHUVLRQ�������������B�
8VLQJ�6FDOD�YHUVLRQ���������-DYD�+RW6SRW�70�����%LW�6HUYHU�90��-DYD������B���6SDUN�FRQWH[W�DYDLODEOH�DV�VF�
VFDOD!�YDO�UGG� �VF�WH[W)LOH��KGIV���QDPHV�MVRQ�����7KLV�LV�D�ORQJ�UXQQLQJ�DSS
VSDUN�VKHOO
&OXVWHU
&OXVWHU�0DQDJHU
'49���1IQ
����
6SDUN�RYHUYLHZ
'ULYHU
([HFXWRU
([HFXWRU:RUG&RXQW�VFDOD
VSDUN�VKHOO
'ULYHU
&OXVWHU
&OXVWHU�0DQDJHU
&38���0HP
����
6SDUN�RYHUYLHZ
'ULYHU
([HFXWRU:RUG&RXQW�VFDOD
([HFXWRU
([HFXWRU([HFXWRU
([HFXWRU
VSDUN�VKHOO
'ULYHU
&OXVWHU
&OXVWHU�0DQDJHU
&38���0HP
����
6SDUN�RYHUYLHZ
'ULYHU
([HFXWRU:RUG&RXQW�VFDOD
([HFXWRU
([HFXWRU([HFXWRU
([HFXWRU
VSDUN�VKHOO
'ULYHU
VSDUN�VKHOO�EHFRPHV�LGOH�EXW�FRQWLQXHV�WR�KROG�RQWR�DOO�FOXVWHU�UHVRXUFHV�
0HDQZKLOH��RWKHU�DSSOLFDWLRQV�FDQQRW�EH�ODXQFKHG�RQ�WKH�FOXVWHU�
6SDUN�UHXVHV�FOXVWHU�UHVRXUFHV
(IILFLHQW�GDWD�VKDULQJ)LQH�JUDLQHG�SDUWLWLRQLQJ/RZ�ODWHQF\�VFKHGXOLQJ
«�EXW�PD\�KROG�RQWR�UHVRXUFHV�IRU�D�ORQJ�WLPH�
5HVRXUFH�&38���0HP�
7LPH
$OORFDWHG8VHG
6WDWLF�DOORFDWLRQ
0RUH�UHVRXUFHV�DOORFDWHG�WKDQ�XVHG
5HVRXUFH�&38���0HP�
7LPH
$OORFDWHG8VHG
'\QDPLF�DOORFDWLRQ
0RUH�HIILFLHQW�XWLOL]DWLRQ�RI�FOXVWHU�UHVRXUFHV
:K\�LV�WKLV�D�6SDUN�SUREOHP"
([HFXWLRQ�PRGHO�LV�GLIIHUHQW�IURP�V\VWHPV�OLNH�05
:K\�LV�WKLV�D�6SDUN�SUREOHP"
([HFXWLRQ�PRGHO�LV�GLIIHUHQW�IURP�V\VWHPV�OLNH�05
6SDUN�UHXVHV�ORQJ�UXQQLQJ�H[HFXWRUV�IRU�VSHHG
05�ODXQFKHV�VKRUW�OLYHG�FRQWDLQHUV�IRU�HDFK�WDVN
'\QDPLF�DOORFDWLRQ
«�DOORZV�\RXU�6SDUN�DSSOLFDWLRQ�WR�VFDOH�WKH�QXPEHU�RI�H[HFXWRUV�XS�DQG�GRZQ�EDVHG�RQ�WKH�ZRUNORDG
,I�H[HFXWRUV�DUH�LGOH��UHPRYH�WKHP�,I�ZH�QHHG�PRUH�H[HFXWRUV��UHTXHVW�WKHP�
6FDOH�XS�SROLF\
5HTXHVW�H[HFXWRUV�ZKHQ�WKHUH�DUH�SHQGLQJ�WDVNV
6FDOH�XS�LQ�PXOWLSOH�URXQGV
([SRQHQWLDO�LQFUHDVH�LQ���H[HFXWRUV�RYHU�WLPH
��H[HFXWRUV
7LPH
6FDOH�XS�SROLF\
([SRQHQWLDO�LQFUHDVH�ZKLOH�WKHUH�DUH�SHQGLQJ�WDVNV
:K\"
Ɣ �6ORZ�VWDUW��LQ�FDVH�ZH�RQO\�QHHG�D�IHZ�PRUH�H[HFXWRUV
Ɣ ([SRQHQWLDO�JURZWK�LQ�FDVH�LW�WXUQV�RXW�WKDW�ZH�QHHG�PDQ\
6FDOH�GRZQ�SROLF\
5HPRYH�H[HFXWRUV�WKDW�KDYH�EHHQ�LGOH�IRU�1�VHFRQGV��H[HFXWRUV
7LPH
LGOH
LGOH
'\QDPLF�DOORFDWLRQ�H[DPSOH��H[HFXWRUV
7LPH
'\QDPLF�DOORFDWLRQ�XVH�FDVHV
/RQJ�UXQQLQJ�(7/�MREV(�J��3DUVLQJ�3%�RI�-621�LQWR�3DUTXHW�LQ�6�
,QWHUDFWLYH�DSSOLFDWLRQV���VHUYHU(�J��6SDUN�VKHOO��2R\DOD�MRE�VHUYHU
$Q\�DSSOLFDWLRQ�ZLWK�ODUJH�VKXIIOHV
'\QDPLF�DOORFDWLRQ�VHWXS
VSDUN�G\QDPLF$OORFDWLRQ�HQDEOHGVSDUN�VKXIIOH�VHUYLFH�HQDEOHG
��2SWLRQDO��VSDUN�G\QDPLF$OORFDWLRQ�PLQ([HFXWRUVVSDUN�G\QDPLF$OORFDWLRQ�PD[([HFXWRUVVSDUN�G\QDPLF$OORFDWLRQ�LQLWLDO([HFXWRUVVSDUN�G\QDPLF$OORFDWLRQ�H[HFXWRU,GOH7LPHRXWVSDUN�G\QDPLF$OORFDWLRQ�VFKHGXOHU%DFNORJ7LPHRXWVSDUN�G\QDPLF$OORFDWLRQ�VXVWDLQHG6FKHGXOHU%DFNORJ7LPHRXW
WUXHWUXH
�
����V�V�V
'\QDPLF�DOORFDWLRQ�VHWXS
$GGLWLRQDOO\��VHW�XS�H[WHUQDO�VKXIIOH�VHUYLFH
)RU�PRUH�GHWDLO�KWWS���VSDUN�DSDFKH�RUJ�GRFV�ODWHVW�MRE�VFKHGXOLQJ�KWPO�G\QDPLF�UHVRXUFH�DOORFDWLRQ
'HPR
&DVH�VWXG\����7%�PHPRU\�ZLWK�������QRGHV
����3%�6��ZDUHKRXVH����3%�3DUTXHW�
'\QDPLF�DOORFDWLRQ�ZLWK�XS�WR��������H[HFXWRUV
(QDEOH�G\QDPLF�DOORFDWLRQ�IRU�DOO�6SDUN�DSSOLFDWLRQV
5XQ�6SDUN�DORQJVLGH�+LYH��3LJ��0DS5HGXFH
8VHG�IRU�DG�KRF�TXHU\�DQG�H[SHULPHQWDWLRQ
�WKLV�WHDP�RQO\
&DVH�VWXG\
�����7%�PHPRU\�ZLWK�������QRGHV
����3%�GDWD�ZDUHKRXVH
8VH�G\QDPLF�DOORFDWLRQ�ZLWK�XS�WR������H[HFXWRUV
3ULPDU\�XVH�FDVHV�DUH�(7/�DQG�64/
5XQ�6SDUN�DORQJVLGH�6WRUP��0DS5HGXFH��3LJ
'\QDPLF�DOORFDWLRQ�VWDWXV
$YDLODEOH�VLQFH�6SDUN������
.QRZQ�LVVXH�LQ�6SDUN��������IL[HG�LQ������
6WDEOH�LQ�6SDUN�������RQZDUGV
&DFKLQJ�VXSSRUW�FRPLQJ�LQ������
0DQXDO�G\QDPLF�DOORFDWLRQ
���5HTXHVW���H[WUD�H[HFXWRUVVF�UHTXHVW([HFXWRUV���
���.LOO�H[HFXWRUV�ZLWK�,'V��������DQG���VF�NLOO([HFXWRUV�6HT������������������
1HZ�GHYHORSHU�$3,�DV�RI�6SDUN����
)XWXUH
6XSSRUW�IRU�0HVRV�PRGH�63$5.�������35�UHDG\�
6XSSRUW�IRU�VWDQGDORQH�PRGH�63$5.�������35�VRRQ�
%HWWHU�VXSSRUW�IRU�FDFKLQJ�63$5.�������35�PHUJHG��
,QWHJUDWLRQ�ZLWK�FORXG�FRPSXWH�SODWIRUP
6XPPDU\
6SDUN�LV�IDVW�EHFDXVH�LW�UHXVHV�FOXVWHU�UHVRXUFHV
6SDUN�KROGV�RQWR�UHVRXUFHV�IRU�D�ORQJ�WLPH
'\QDPLF�DOORFDWLRQ�DOORZV�\RX�WR�EHQHILW�IURP�WKH�IRUPHU�ZLWKRXW�VXIIHULQJ�WKH�ODWWHU
7KDQN�\RX�+!/"4ȯ!�1��/& (0ǽ ,*