そのデータフロー nifiで楽にしてあげましょう

ŕšƕƲƏƞưƲęĬĒĬŜ�ŠŐśłŌŦŐŮńKoji KawamuraHortonworks Software EngineerJuly 27, 2016

2 ©HortonworksInc.2011–2016.AllRightsReserved

ÂV·�

��\  üŇŴŨŰ ōńőý

- Hortonworks DataFlow Team – Software Engineer- ijokarumawak- kkawamura hortonworks.comÿ ĔĤħııIJŬęıĞĜėşŞ�wŸƠƭƅƲƉƫƱŵ §ŐŖƉƋƔƦé©ŵº÷ĽĎIJĤĦīĨùęĬĒĬňĚĞĞŝŐś�éŏųśêŪşŃòŠƆƱƖƭƝƩƲƖŐŖƠƮƭƃŽƋƖňH�ŝşűĔıijĵıİķıijĭĴŠŽƱƊƙŸ�Ŝ�±Ľ~�)(ľ

ÿ ĉ�ŇŰ�ųśĎIJĤĦīĨùęĬĒĬƆƥƒƏƲŠēĬĵƆƤƱƗfŃļ

3 ©HortonworksInc.2011–2016.AllRightsReserved

AgendaHDF, Apache NiFiŝŢ

źƱƋƖƲƮ|�

Step 1. ƕƲƏŵîũŲ

Step 2. ƕƲƏŵ%UŒŲ

Step 3. ƕƲƏŵ�öŒŲ

4 ©HortonworksInc.2011–2016.AllRightsReserved

ƕƲƏš�µ

ƕƲƏš�¥ŝ��

ƕƲƏš/b

DataFlowsš�¨<Ƴ´+Ľ�¯

ƕƲƏƞưƲ

5 ©HortonworksInc.2011–2016.AllRightsReserved

NìšƕƲƏƞưƲƳÊï

ƕƲƏš�µ

ƕƲƏš�¥ŝ��

ƕƲƏš/b

ƕƲƏš�µƕƲƏš�µ

ƕƲƏš�µ

ƕƲƏš�µ

ƕƲƏš/b

ƕƲƏš/b

ƕƲƏš/b

ƕƲƏƞưƲ

6 ©HortonworksInc.2011–2016.AllRightsReserved

TheConnectedDataPlatform

7 ©HortonworksInc.2011–2016.AllRightsReserved

»3ŏųŖƠưƍƋŝ!c

�êşŸƲƂƔƃƑƨ

H O R T O N W O R K S D A T A F L O W ş Ő

H O R T O N W O R K S D A T A F L O W ł ű

/űÛŧ

ƋƃƭƠƖ

ƧƒƍƲƊƱƄ

ƋƃƭƠƖ

HORTONWORKS DATAFLOW

ŸƲƂƔƃƑƨŵ�ä*�Ū&£ªşƕƲƏ.îƔƃƚưƊƲŠůűƆƋƖŝÊïŏŵ"�

&£ªşâ§ŵ�ÐƕƲƏš�$Ľ�$ĽÞ�ĽDtŵƭŸƮƏźƦŠ!c

�ß@«šs§ƀƋƏƤźƌŏųŖÊxƉƋƔƦŤš�Khŵqë

8 ©HortonworksInc.2011–2016.AllRightsReserved

N½šłŲâ§�&h

ƝƊƩŸƮƪƲƈƲźƱƏƲƞżźƋƗƬƒƄ&ƗưƒƠŠůŲ&£ªŜ�vşu�,�ƞŹƲƗƜƒƃƕƲƏƞưƲŵƭŸƮƏźƦŠéJĽ��ĽÕyĽ�é

ƢƭƩƲƦŝW?XŤšä3FRŵ8ŴœĽłŰŭŲƕƲƏŠQe

źơƱƖƯơƮšƕƲƏ��ƁƜƘƱƋĽƆƱƠƬźŸƱƋĽƕƲƏÑ�

ƕƲƏšƍƂƩŸş/bŝÙÞ!cŏųŖƕƲƏ��ŝånªşƕƲƏ�Ã*šŖũš¶�hšøŃ�1*

Powered by Apache NiFi

9 ©HortonworksInc.2011–2016.AllRightsReserved

ã-8YêŠNSAŠůřśé©

ŀNSAšźƚơƲƏƲŢĽÁŅŰųŲ�Ŝ�Ū;ñş=OL��íš8õš�æŠ/ű¹ŶŗŁ

ŀ7�ŽƱƏƲƠƬźƌŢĽōųŵ §Őś>¥ªŠðųŖƇźƖŇŰšiA

ƞưƲŵÜàŠ!cĽ³¥Ľ��ŐĽ¼3ªş¢�ÔÖŵ¦ŧ�ŒōŝňŜʼnŲŁ

-- NSAƕŹƯƃƏƲLinda L. Burger�

=OL��íSňé©ŐŖNiFi

10 ©HortonworksInc.2011–2016.AllRightsReserved

ƭŸƮƏźƦƕƲƏưƊƋƔŹƃƋš»3ªŇŚźƱƏƬƃƔŹƟş!c

HTTP

Syslog

ƧƲƮ

HTML

źƧƲƊ

ƛƒƉƩ �1*

o�

ƔƲƮƤƲƊ

�

ÊÉ NÇ

ƋƂƨƱ

ƊſŽƱƭƒƑ

¿t

D_�#

Dt

HL7

FTP

UDP

XML

SFTP

ƆƱƔƱƓšƮƲƔŹƱƄ

ƆƱƔƂƋƖšƮƲƔŹƱƄ

ƔƂƋƖšƮƲƔŹƱƄ

ƯƲƖš!c

×Åš�w

11 ©HortonworksInc.2011–2016.AllRightsReserved

Ňŝ� ŕńşšŜĽŦœŢ řśŧůńú

12 ©HortonworksInc.2011–2016.AllRightsReserved

AgendaHDF, Apache NiFiŝŢ

źƱƋƖƲƮ|�

Step 1. ƕƲƏŵîũŲ

Step 2. ƕƲƏŵ%UŒŲ

Step 3. ƕƲƏŵ�öŒŲ

13 ©HortonworksInc.2011–2016.AllRightsReserved

źƱƋƖƲƮ|�

à Apache NiFi +ŜŪ

à HDPŝƍƒƖŜŪ

à DockerźƧƲƊŪłŲšŜ

à Apache NiFi ƎƲƋŇŰƝƮƗŐśŪĚĖúùüŇşű�êŇŇűŦŒŋŞý

14 ©HortonworksInc.2011–2016.AllRightsReserved

ĎIJĤĦīĨùęĬĒĬ źƱƋƖƲƮ|�Čù+ŜmÚŠÒŐŖŃ|5ŋ

ăĀ ƜźƘƭŵƐŻƱưƲƗ ĢăģùŐśTé

ĄĀ Ø'$ ./bin/nifi.sh start

ąĀ ƟƬŻƈŇŰŸƃƍƋīĵĵIJČāāĮıĦĤĮīıĴĵČĊĂĊĂāİĬĩĬ

ĆĀ ƇƲƝƋŝŐśźƱƋƖƲƮŒŲB3$ ./bin/nifi.sh install

ćĀ ÏMƞŷźƮŢōō ĢĄģĀāĦıİĩāİĬĩĬĀIJijıIJĨijĵĬĨĴ

[1] https://nifi.apache.org/download.html[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties

15 ©HortonworksInc.2011–2016.AllRightsReserved

HDF źƱƋƖƲƮ|�ČùĔĐěŝƍƒƖŜņÒŐŐŖŃ|5ŋ

ĔĐěùĞĤİħĥıĸŵƐŻƱưƲƗ ĢăģùŐśØ'ĽĎįĥĤijĬŇŰęĬĒĬƇƲƝƋŵÝ%ĽƕƞžƮƖŗŝċĂċĂƣƲƖŜØ'īĵĵIJČāāĮıĦĤĮīıĴĵČċĂċĂāİĬĩĬ ŠŸƃƍƋ

�ČùġĨijĴĬıİŢĔĐĒšġĨijĴĬıİ

ĔĐĒùČùęĬĒĬăĀăĸùČùĂĀćăĀĄĸùČùĂĀĈĄĀĂĸùČùăĀĂ

[1] http://hortonworks.com/downloads/#dataflow

16 ©HortonworksInc.2011–2016.AllRightsReserved

´+şƑƩƲƖƭŸƮE`şƕƲƏƎƲƋŠ¾ňŲšŗňĽęĬĒĬÂŜŢŞŶşōŝňŜʼnŲšŇ

17 ©HortonworksInc.2011–2016.AllRightsReserved

AgendaHDF, Apache NiFiŝŢ

źƱƋƖƲƮ|�

Step 1. ƕƲƏŵîũŲ

Step 2. ƕƲƏŵ%UŒŲ

Step 3. ƕƲƏŵ�öŒŲ

18 ©HortonworksInc.2011–2016.AllRightsReserved

Step 1 : ƕƲƏŵîũŲ

19 ©HortonworksInc.2011–2016.AllRightsReserved

ōōňƣźƱƖ

à ƕƲƏš�ųŵĐĤĵĤùĒĮıķŝŃŃĽ�ųŲƕƲƏŵĒĮıķùĒĬĮĨŝŃń

à ĒĮıķùĒĬĮĨŢĶĶĬħþùďıİĵĨİĵþùĎĵĵijĬĥĶĵĨŵpŚ

à Flow Fileŵ�řŖű0�ŐŖű%UŐŖűÙÞŐŖű�#ŐŖűƤƲƊŐŖűŒŲšňěijıĦĨĴĴıij

à ProcessorŢ+�šƏƋƃŠ¡*ŐśƕƈźƱŏųśŃŲĽ}KšěijıĦĨĴĴıijŵńŦŊŚşŌŲšňçË

à ěijıĦĨĴĴıijšęĤįĨŠŵŒŲšŇ�ŃśņŊŝaĿ�ŇűŬŒŃ

à ProcessoršÏMŠ��ňłŲŝěijıĦĨĴĴıijŢĴĵĤijĵŜʼnşŃĽÆŘ­ŃśěijıĦĨĴĴıijŠÈ°ŏųŲƧƒƍƲƊŵÌůń

à Processor4CŵŚşŌŲšňĝĨĮĤĵĬıİĴīĬIJ

à RelationshipŢ�šěijıĦĨĴĴıijŠŚşŌŲŇĽÂ'Ŝ¸�üĎĶĵıù ğĨijįĬİĤĵĨýŏŔŲdËňłŲ

20 ©HortonworksInc.2011–2016.AllRightsReserved

AgendaHDF, Apache NiFiŝŢ

źƱƋƖƲƮ|�

Step 1. ƕƲƏŵîũŲ

Step 2. ƕƲƏŵ%UŒŲ

Step 3. ƕƲƏŵ�öŒŲ

21 ©HortonworksInc.2011–2016.AllRightsReserved

Step 2 : ƕƲƏŵ%UŒŲ

logAttributes^([\d]{4}-[\d]{2}-[\d]{2}

[\d]{2}:[\d]{2}:[\d]{2},[\d]{3}) ([A-Z]+) (.+)$

22 ©HortonworksInc.2011–2016.AllRightsReserved

ōōňƣźƱƖ

à Data FlowňkèŐśʼnŖŰĽěijıĦĨĴĴùēijıĶIJŠR�ŋŐśƧƱƔŵ�ŠŐůń

à Process GroupêšƕƲƏÙÞŢĕİIJĶĵāĚĶĵIJĶĵ IJıijĵŵ ń

à Flow FileŢÊxÇšƔƂƋƖŵ6ŨōŝňŜʼnŲĽ�ÇœŚƮƲƔŹƱƄŐŖŃB3ŢĞIJĮĬĵğĨĸĵŵ�§Őůń

à ŗňŐŇŐĽ�ZŠFʼnşƞŷźƮŵĞIJĮĬĵğĨĸĵŜ�ÇŎŝŠ�#ŐśŐŦńŝĚĶĵĚĩĘĨįıijĹň©¦ŒŲB3ňłŲšŜĽE�ŠŒŲšňÄŃ

à TailFileŵ ńB3ŠƞŷźƮňưƲƔźƖŒŲB3ŢĽĻĝıĮĮĬİĪùĒĬĮĨİĤįĨùěĤĵĵĨijİĻŵdœÏMŐůń

à ƔƂƋƖ_]šFlow FilešďıİĵĨİĵùü�zĽĥıħĹýŵÎ�ŒŲB3ŢđĸĵijĤĦĵğĨĸĵŵ ņń

à Dynamic PropertyŢƪƲƈň�jŠÝ%ŜʼnŲěijıĦĨĴĴıijšÏMó¬ĽđĸĵijĤĦĵğĨĸĵŜŢōųŠ�ÍȤŵMÀŒŲ

23 ©HortonworksInc.2011–2016.AllRightsReserved

NiFišƕƲƏŵƃƭŸŒŲŠŢč

NiFiŵ�� ûùĀāĥĬİāİĬĩĬĀĴī ĴĵıIJ*_repository ŢƕƲƏš�³[Ľ�ËşŰƕŹƯƃƖƭ"ëŐśŐŦřśŪÄŃlogs, run, state, workŪ"ëŐśF�GĠĕŜô^řś�kŐŖĐĤĵĤùĒĮıķŢĀāĦıİĩāĩĮıķĀĸįĮĀĪĺ Š�KŏųśŃŲōšĩĮıķĀĸįĮŗŋ�řśŃųţF�G

�ČùłŊŦŜé©�šÓŜŒůú

24 ©HortonworksInc.2011–2016.AllRightsReserved

AgendaHDF, Apache NiFiŝŢ

źƱƋƖƲƮ|�

Step 1. ƕƲƏŵîũŲ

Step 2. ƕƲƏŵ%UŒŲ

Step 3. ƕƲƏŵ�öŒŲ

25 ©HortonworksInc.2011–2016.AllRightsReserved

Step 3 : ƕƲƏŵ�öŒŲ

26 ©HortonworksInc.2011–2016.AllRightsReserved

ōōňƣźƱƖ

à ęĬĒĬšđĸIJijĨĴĴĬıİùėĤİĪĶĤĪĨŵ ŃōşŒ Ģăģ

à ĝıĶĵĨĚİĎĵĵijĬĥĶĵĨšůńŠĽĐĹİĤįĬĦùěijıIJĨijĵĹŝá'ŐśĽĝĨĮĤĵĬıİĴīĬIJŪ�jŠÝ%ŜʼnŲěijıĦĨĴĴıijňłŲ

à ĐĤĵĤĒĮıķšâ9ŵéJŐŖaŠĽ�æŗŋ��Ľ{ÍƮƲƖŵ�kŐŖűŜʼnŲ

à gŃßűŠĒĮıķùĒĬĮĨšƮƲƔŹƱƄňŜʼnųţĽaŢņIʼnşěijıĦĨĴĴıijŠŚşŌŲŗŋĽdËşƕƲƏŵĽdËşBlŤ üĒĬĮĨþùďĮıĶħùĴĵıijĤĪĨþùĐĤĵĤĥĤĴĨþùĔĐĒĞĀĀĀùĨĵĦý

ĢăģùīĵĵIJĴČāāİĬĩĬĀĤIJĤĦīĨĀıijĪāħıĦĴāİĬĩĬÿħıĦĴāīĵįĮāĨĸIJijĨĴĴĬıİÿĮĤİĪĶĤĪĨÿĪĶĬħĨĀīĵįĮ

27 ©HortonworksInc.2011–2016.AllRightsReserved

/bŐŖƕƲƏš�§�ČùĞĮĤĦĭŠƧƒƍƲƊß®

28 ©HortonworksInc.2011–2016.AllRightsReserved

Provenance� ŜŒú

29 ©HortonworksInc.2011–2016.AllRightsReserved

�:ŐūťřŖ�PŝĽ�aŐūťűŖŃ�P

�:ŐūťřŖ

źƱƋƖƲƮ|�Ľ 2²ƠưƍƒƇƠưƍƋƄƮƲƠĽ ƭƯƲƉƫƱĽ ƞưƲƞŷźƮĽ ŸƖƭƝƩƲƖĽƠƮƲƟƘƱƋ

�a�rűŐŖŃ

ÏMƞŷźƮĽ ęĤij Ľ ƕƲƏŽƱƭƒƑĽ ŽƃƋƠƯƒƉƫƱĽ ƞưƲƆƱƖưƲƬĽ ƃƬƋƏĽ ƋƔƲƏƋĽ ƆƱƖưƲƬƇƲƝƋĽ ƯƣƲƔŹƱƄƏƋƃĽ ƞưƲƞŷźƮƭƣƊƖƭĽƆƱƔƱƖƭƣƊƖƭĽ ƠƮƲƟƘƱƋƭƣƊƖƭĽ ƜƒƃƠƯƒƉƨĽƠƬźſƭƏźƌĽ ƔƱƠƯƲƖĽ ƍƂƩƭƔŹĽ ƤƮƑƔƘƱƉƲ Ľ ĞĬĵĨÿĵıÿĞĬĵĨùļùĨĵĦ

30 ©HortonworksInc.2011–2016.AllRightsReserved

ThankYou