そのデータフロー nifiで楽にしてあげましょう
TRANSCRIPT
ŕšƕƲƏƞưƲęĬĒĬŜ�ŠŐśłŌŦŐŮńKoji KawamuraHortonworks Software EngineerJuly 27, 2016
2 ©HortonworksInc.2011–2016.AllRightsReserved
ÂV·�
��\ üŇŴŨŰ ōńőý
- Hortonworks DataFlow Team – Software Engineer- ijokarumawak- kkawamura hortonworks.comÿ ĔĤħııIJŬęıĞĜėşŞ�wŸƠƭƅƲƉƫƱŵ §ŐŖƉƋƔƦé©ŵº÷ĽĎIJĤĦīĨùęĬĒĬňĚĞĞŝŐś�éŏųśêŪşŃòŠƆƱƖƭƝƩƲƖŐŖƠƮƭƃŽƋƖňH�ŝşűĔıijĵıİķıijĭĴŠŽƱƊƙŸ�Ŝ�±Ľ~�)(ľ
ÿ ĉ�ŇŰ�ųśĎIJĤĦīĨùęĬĒĬƆƥƒƏƲŠēĬĵƆƤƱƗfŃļ
3 ©HortonworksInc.2011–2016.AllRightsReserved
AgendaHDF, Apache NiFiŝŢ
źƱƋƖƲƮ|�
Step 1. ƕƲƏŵîũŲ
Step 2. ƕƲƏŵ%UŒŲ
Step 3. ƕƲƏŵ�öŒŲ
4 ©HortonworksInc.2011–2016.AllRightsReserved
ƕƲƏš�µ
ƕƲƏš�¥ŝ��
ƕƲƏš/b
DataFlowsš�¨<Ƴ´+Ľ�¯
ƕƲƏƞưƲ
5 ©HortonworksInc.2011–2016.AllRightsReserved
NìšƕƲƏƞưƲƳÊï
ƕƲƏš�µ
ƕƲƏš�¥ŝ��
ƕƲƏš/b
ƕƲƏš�µƕƲƏš�µ
ƕƲƏš�µ
ƕƲƏš�µ
ƕƲƏš/b
ƕƲƏš/b
ƕƲƏš/b
ƕƲƏƞưƲ
6 ©HortonworksInc.2011–2016.AllRightsReserved
TheConnectedDataPlatform
7 ©HortonworksInc.2011–2016.AllRightsReserved
»3ŏųŖƠưƍƋŝ!c
�êşŸƲƂƔƃƑƨ
H O R T O N W O R K S D A T A F L O W ş Ő
H O R T O N W O R K S D A T A F L O W ł ű
/űÛŧ
ƋƃƭƠƖ
ƧƒƍƲƊƱƄ
ƋƃƭƠƖ
HORTONWORKS DATAFLOW
ŸƲƂƔƃƑƨŵ�ä*�Ū&£ªşƕƲƏ.îƔƃƚưƊƲŠůűƆƋƖŝÊïŏŵ"�
&£ªşâ§ŵ�ÐƕƲƏš�$Ľ�$ĽÞ�ĽDtŵƭŸƮƏźƦŠ!c
�ß@«šs§ƀƋƏƤźƌŏųŖÊxƉƋƔƦŤš�Khŵqë
8 ©HortonworksInc.2011–2016.AllRightsReserved
N½šłŲâ§�&h
ƝƊƩŸƮƪƲƈƲźƱƏƲƞżźƋƗƬƒƄ&ƗưƒƠŠůŲ&£ªŜ�vşu�,�ƞŹƲƗƜƒƃƕƲƏƞưƲŵƭŸƮƏźƦŠéJĽ��ĽÕyĽ�é
ƢƭƩƲƦŝW?XŤšä3FRŵ8ŴœĽłŰŭŲƕƲƏŠQe
źơƱƖƯơƮšƕƲƏ��ƁƜƘƱƋĽƆƱƠƬźŸƱƋĽƕƲƏÑ�
ƕƲƏšƍƂƩŸş/bŝÙÞ!cŏųŖƕƲƏ��ŝånªşƕƲƏ�Ã*šŖũš¶�hšøŃ�1*
Powered by Apache NiFi
9 ©HortonworksInc.2011–2016.AllRightsReserved
ã-8YêŠNSAŠůřśé©
ŀNSAšźƚơƲƏƲŢĽÁŅŰųŲ�Ŝ�Ū;ñş=OL��íš8õš�æŠ/ű¹ŶŗŁ
ŀ7�ŽƱƏƲƠƬźƌŢĽōųŵ §Őś>¥ªŠðųŖƇźƖŇŰšiA
ƞưƲŵÜàŠ!cĽ³¥Ľ��ŐĽ¼3ªş¢�ÔÖŵ¦ŧ�ŒōŝňŜʼnŲŁ
-- NSAƕŹƯƃƏƲLinda L. Burger�
=OL��íSňé©ŐŖNiFi
10 ©HortonworksInc.2011–2016.AllRightsReserved
ƭŸƮƏźƦƕƲƏưƊƋƔŹƃƋš»3ªŇŚźƱƏƬƃƔŹƟş!c
HTTP
Syslog
ƧƲƮ
HTML
źƧƲƊ
ƛƒƉƩ �1*
o�
ƔƲƮƤƲƊ
�
ÊÉ NÇ
ƋƂƨƱ
ƊſŽƱƭƒƑ
¿t
D_�#
Dt
HL7
FTP
UDP
XML
SFTP
ƆƱƔƱƓšƮƲƔŹƱƄ
ƆƱƔƂƋƖšƮƲƔŹƱƄ
ƔƂƋƖšƮƲƔŹƱƄ
ƯƲƖš!c
×Åš�w
11 ©HortonworksInc.2011–2016.AllRightsReserved
Ňŝ� ŕńşšŜĽŦœŢ řśŧůńú
12 ©HortonworksInc.2011–2016.AllRightsReserved
AgendaHDF, Apache NiFiŝŢ
źƱƋƖƲƮ|�
Step 1. ƕƲƏŵîũŲ
Step 2. ƕƲƏŵ%UŒŲ
Step 3. ƕƲƏŵ�öŒŲ
13 ©HortonworksInc.2011–2016.AllRightsReserved
źƱƋƖƲƮ|�
à Apache NiFi +ŜŪ
à HDPŝƍƒƖŜŪ
à DockerźƧƲƊŪłŲšŜ
à Apache NiFi ƎƲƋŇŰƝƮƗŐśŪĚĖúùüŇşű�êŇŇűŦŒŋŞý
14 ©HortonworksInc.2011–2016.AllRightsReserved
ĎIJĤĦīĨùęĬĒĬ źƱƋƖƲƮ|�Čù+ŜmÚŠÒŐŖŃ|5ŋ
ăĀ ƜźƘƭŵƐŻƱưƲƗ ĢăģùŐśTé
ĄĀ Ø'$ ./bin/nifi.sh start
ąĀ ƟƬŻƈŇŰŸƃƍƋīĵĵIJČāāĮıĦĤĮīıĴĵČĊĂĊĂāİĬĩĬ
ĆĀ ƇƲƝƋŝŐśźƱƋƖƲƮŒŲB3$ ./bin/nifi.sh install
ćĀ ÏMƞŷźƮŢōō ĢĄģĀāĦıİĩāİĬĩĬĀIJijıIJĨijĵĬĨĴ
[1] https://nifi.apache.org/download.html[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
15 ©HortonworksInc.2011–2016.AllRightsReserved
HDF źƱƋƖƲƮ|�ČùĔĐěŝƍƒƖŜņÒŐŐŖŃ|5ŋ
ĔĐěùĞĤİħĥıĸŵƐŻƱưƲƗ ĢăģùŐśØ'ĽĎįĥĤijĬŇŰęĬĒĬƇƲƝƋŵÝ%ĽƕƞžƮƖŗŝċĂċĂƣƲƖŜØ'īĵĵIJČāāĮıĦĤĮīıĴĵČċĂċĂāİĬĩĬ ŠŸƃƍƋ
�ČùġĨijĴĬıİŢĔĐĒšġĨijĴĬıİ
ĔĐĒùČùęĬĒĬăĀăĸùČùĂĀćăĀĄĸùČùĂĀĈĄĀĂĸùČùăĀĂ
[1] http://hortonworks.com/downloads/#dataflow
16 ©HortonworksInc.2011–2016.AllRightsReserved
´+şƑƩƲƖƭŸƮE`şƕƲƏƎƲƋŠ¾ňŲšŗňĽęĬĒĬÂŜŢŞŶşōŝňŜʼnŲšŇ
17 ©HortonworksInc.2011–2016.AllRightsReserved
AgendaHDF, Apache NiFiŝŢ
źƱƋƖƲƮ|�
Step 1. ƕƲƏŵîũŲ
Step 2. ƕƲƏŵ%UŒŲ
Step 3. ƕƲƏŵ�öŒŲ
18 ©HortonworksInc.2011–2016.AllRightsReserved
Step 1 : ƕƲƏŵîũŲ
19 ©HortonworksInc.2011–2016.AllRightsReserved
ōōňƣźƱƖ
à ƕƲƏš�ųŵĐĤĵĤùĒĮıķŝŃŃĽ�ųŲƕƲƏŵĒĮıķùĒĬĮĨŝŃń
à ĒĮıķùĒĬĮĨŢĶĶĬħþùďıİĵĨİĵþùĎĵĵijĬĥĶĵĨŵpŚ
à Flow Fileŵ�řŖű0�ŐŖű%UŐŖűÙÞŐŖű�#ŐŖűƤƲƊŐŖűŒŲšňěijıĦĨĴĴıij
à ProcessorŢ+�šƏƋƃŠ¡*ŐśƕƈźƱŏųśŃŲĽ}KšěijıĦĨĴĴıijŵńŦŊŚşŌŲšňçË
à ěijıĦĨĴĴıijšęĤįĨŠŵŒŲšŇ�ŃśņŊŝaĿ�ŇűŬŒŃ
à ProcessoršÏMŠ��ňłŲŝěijıĦĨĴĴıijŢĴĵĤijĵŜʼnşŃĽÆŘŃśěijıĦĨĴĴıijŠÈ°ŏųŲƧƒƍƲƊŵÌůń
à Processor4CŵŚşŌŲšňĝĨĮĤĵĬıİĴīĬIJ
à RelationshipŢ�šěijıĦĨĴĴıijŠŚşŌŲŇĽÂ'Ŝ¸�üĎĶĵıù ğĨijįĬİĤĵĨýŏŔŲdËňłŲ
20 ©HortonworksInc.2011–2016.AllRightsReserved
AgendaHDF, Apache NiFiŝŢ
źƱƋƖƲƮ|�
Step 1. ƕƲƏŵîũŲ
Step 2. ƕƲƏŵ%UŒŲ
Step 3. ƕƲƏŵ�öŒŲ
21 ©HortonworksInc.2011–2016.AllRightsReserved
Step 2 : ƕƲƏŵ%UŒŲ
logAttributes^([\d]{4}-[\d]{2}-[\d]{2}
[\d]{2}:[\d]{2}:[\d]{2},[\d]{3}) ([A-Z]+) (.+)$
22 ©HortonworksInc.2011–2016.AllRightsReserved
ōōňƣźƱƖ
à Data FlowňkèŐśʼnŖŰĽěijıĦĨĴĴùēijıĶIJŠR�ŋŐśƧƱƔŵ�ŠŐůń
à Process GroupêšƕƲƏÙÞŢĕİIJĶĵāĚĶĵIJĶĵ IJıijĵŵ ń
à Flow FileŢÊxÇšƔƂƋƖŵ6ŨōŝňŜʼnŲĽ�ÇœŚƮƲƔŹƱƄŐŖŃB3ŢĞIJĮĬĵğĨĸĵŵ�§Őůń
à ŗňŐŇŐĽ�ZŠFʼnşƞŷźƮŵĞIJĮĬĵğĨĸĵŜ�ÇŎŝŠ�#ŐśŐŦńŝĚĶĵĚĩĘĨįıijĹň©¦ŒŲB3ňłŲšŜĽE�ŠŒŲšňÄŃ
à TailFileŵ ńB3ŠƞŷźƮňưƲƔźƖŒŲB3ŢĽĻĝıĮĮĬİĪùĒĬĮĨİĤįĨùěĤĵĵĨijİĻŵdœÏMŐůń
à ƔƂƋƖ_]šFlow FilešďıİĵĨİĵùü�zĽĥıħĹýŵÎ�ŒŲB3ŢđĸĵijĤĦĵğĨĸĵŵ ņń
à Dynamic PropertyŢƪƲƈň�jŠÝ%ŜʼnŲěijıĦĨĴĴıijšÏMó¬ĽđĸĵijĤĦĵğĨĸĵŜŢōųŠ�ÍȤŵMÀŒŲ
23 ©HortonworksInc.2011–2016.AllRightsReserved
NiFišƕƲƏŵƃƭŸŒŲŠŢč
NiFiŵ�� ûùĀāĥĬİāİĬĩĬĀĴī ĴĵıIJ*_repository ŢƕƲƏš�³[Ľ�ËşŰƕŹƯƃƖƭ"ëŐśŐŦřśŪÄŃlogs, run, state, workŪ"ëŐśF�GĠĕŜô^řś�kŐŖĐĤĵĤùĒĮıķŢĀāĦıİĩāĩĮıķĀĸįĮĀĪĺ Š�KŏųśŃŲōšĩĮıķĀĸįĮŗŋ�řśŃųţF�G
�ČùłŊŦŜé©�šÓŜŒůú
24 ©HortonworksInc.2011–2016.AllRightsReserved
AgendaHDF, Apache NiFiŝŢ
źƱƋƖƲƮ|�
Step 1. ƕƲƏŵîũŲ
Step 2. ƕƲƏŵ%UŒŲ
Step 3. ƕƲƏŵ�öŒŲ
25 ©HortonworksInc.2011–2016.AllRightsReserved
Step 3 : ƕƲƏŵ�öŒŲ
26 ©HortonworksInc.2011–2016.AllRightsReserved
ōōňƣźƱƖ
à ęĬĒĬšđĸIJijĨĴĴĬıİùėĤİĪĶĤĪĨŵ ŃōşŒ Ģăģ
à ĝıĶĵĨĚİĎĵĵijĬĥĶĵĨšůńŠĽĐĹİĤįĬĦùěijıIJĨijĵĹŝá'ŐśĽĝĨĮĤĵĬıİĴīĬIJŪ�jŠÝ%ŜʼnŲěijıĦĨĴĴıijňłŲ
à ĐĤĵĤĒĮıķšâ9ŵéJŐŖaŠĽ�æŗŋ��Ľ{ÍƮƲƖŵ�kŐŖűŜʼnŲ
à gŃßűŠĒĮıķùĒĬĮĨšƮƲƔŹƱƄňŜʼnųţĽaŢņIʼnşěijıĦĨĴĴıijŠŚşŌŲŗŋĽdËşƕƲƏŵĽdËşBlŤ üĒĬĮĨþùďĮıĶħùĴĵıijĤĪĨþùĐĤĵĤĥĤĴĨþùĔĐĒĞĀĀĀùĨĵĦý
ĢăģùīĵĵIJĴČāāİĬĩĬĀĤIJĤĦīĨĀıijĪāħıĦĴāİĬĩĬÿħıĦĴāīĵįĮāĨĸIJijĨĴĴĬıİÿĮĤİĪĶĤĪĨÿĪĶĬħĨĀīĵįĮ
27 ©HortonworksInc.2011–2016.AllRightsReserved
/bŐŖƕƲƏš�§�ČùĞĮĤĦĭŠƧƒƍƲƊß®
28 ©HortonworksInc.2011–2016.AllRightsReserved
Provenance� ŜŒú
29 ©HortonworksInc.2011–2016.AllRightsReserved
�:ŐūťřŖ�PŝĽ�aŐūťűŖŃ�P
�:ŐūťřŖ
źƱƋƖƲƮ|�Ľ 2²ƠưƍƒƇƠưƍƋƄƮƲƠĽ ƭƯƲƉƫƱĽ ƞưƲƞŷźƮĽ ŸƖƭƝƩƲƖĽƠƮƲƟƘƱƋ
�a�rűŐŖŃ
ÏMƞŷźƮĽ ęĤij Ľ ƕƲƏŽƱƭƒƑĽ ŽƃƋƠƯƒƉƫƱĽ ƞưƲƆƱƖưƲƬĽ ƃƬƋƏĽ ƋƔƲƏƋĽ ƆƱƖưƲƬƇƲƝƋĽ ƯƣƲƔŹƱƄƏƋƃĽ ƞưƲƞŷźƮƭƣƊƖƭĽƆƱƔƱƖƭƣƊƖƭĽ ƠƮƲƟƘƱƋƭƣƊƖƭĽ ƜƒƃƠƯƒƉƨĽƠƬźſƭƏźƌĽ ƔƱƠƯƲƖĽ ƍƂƩƭƔŹĽ ƤƮƑƔƘƱƉƲ Ľ ĞĬĵĨÿĵıÿĞĬĵĨùļùĨĵĦ
30 ©HortonworksInc.2011–2016.AllRightsReserved
ThankYou