using solr to search and analyze logs
DESCRIPTION
Many of us tend to hate or simply ignore logs, and rightfully so: they’re typically hard to find, difficult to handle, and are cryptic to the human eye. But can we make logs more valuable and more usable if we index them in Solr, so we can search and run real-time statistics on them? Indeed we can, and in this session you’ll learn how to make that happen. In the first part of the session we’ll explain why centralized logging is important, what valuable information one can extract from logs, and we’ll introduce the leading tools from the logging ecosystems everyone should be aware of - from syslog and log4j to LogStash and Flume. In the second part we’ll teach you how to use these tools in tandem with Solr. We’ll show how to use Solr in a SolrCloud setup to index large volumes of logs continuously and efficiently. Then, we'll look at how to scale the Solr cluster as your data volume grows. Finally, we'll see how you can parse your unstructured logs and convert them to nicely structured Solr documents suitable for analytical queries.TRANSCRIPT
1hQ[O�/]Yg�j]�/I<gEP�<[G�
�[<YsvI� ]Oh
.<Gk��PI]gOPI
³g<GkÁOPI]gOPI³hIZ<jIrj
�Y<hjQEhI<gEP��+�
hshY]OgIEIQpIg
]OhI[I
�QD<[<
hshY]OG
]Ohj<hP
7P<j�<D]kj �
GINQ[Q[O�<[G�P<[GYQ[O�Y]Oh�Q[�OI[Ig<Y
�hIjh�]N�j]]Yh�j]�hI[G�Y]Oh�j]
+IgN]gZ<[EI�jk[Q[O�<[G�/]Yg Y]kG
hshY]O
�INQ[Q[O�<[G��<[GYQ[O� ]Oh¥hj]gs�jQZI�¦
hshY]O
hshY]O
hshY]O
�
.IfkQgIZI[jh
¦�7P<j�h�qg][O�
KWWS���HGG\VXDLE�FRP�ZS�FRQWHQW�XSORDGV���������.H\ZRUG�LFRQ�SQJ
¥���������������N]g�GIDkOOQ[O¦
+g]DYIZ
Y]]]]]jh�]N�ZIhh<OIh�E]ZQ[O�Q[
KWWS���ZZZ�VFLHQFHVXUYLYDOEORJ�FRP�JHWWLQJ�SXEOLVKHG�XQILQLVKHG�PDQXVFULSWVB����
/]YpIG�qQjP�[]�Q[GIrQ[O
�10
�Y<hjQEhI<gEP
.IfkQgIZI[jh
¦�7P<j�h�qg][O���䘟
æ�7P<j�qQYY�O]�qg][O�
�������������¥hj<jh¦
+<ghQ[O�.<q� ]Oh
�10
ZQEXIs�Z]khI�ÂÁ
khIg QjIZ jQZI
hjQYY�hY]q N]gZ<j�EP<[OIh
+<ghQ[O�.<q� ]Oh
�10
ZQEXIs�Z]khI�Á�ÂÁ
<GG�Igg]g�E]GI
hjQYY�hY]q N]gZ<j�EP<[OIh
�<EIjh�� ]OOQ[O�Q[��/$"
ÃÁÂÄ�ÂÂ�ÁÇ«�ZQEXIs�Z]khI
©��±G<jI±��±ÃÁÂÄ�ÂÂ�ÁDZ���±ZIhh<OI±��±ZQEXIs�Z]khI±ª
�<EIjh�� ]OOQ[O�Q[��/$"
ÃÁÂÄ�ÂÂ�ÁÇ«�³EII�©±khIg±��±ZQEXIs±ª
©��±G<jI±��±ÃÁÂÄ�ÂÂ�ÁDZ���±khIg±��±ZQEXIs±ª
ÃÁÂÄ�ÂÂ�ÁÇ«�ZQEXIs�Z]khI
©��±G<jI±��±ÃÁÂÄ�ÂÂ�ÁDZ���±ZIhh<OI±��±ZQEXIs�Z]khI±ª
.IfkQgIZI[jh
¦�7P<j�h�qg][O���䘟
æ�7P<j�qQYY�O]�qg][O��䘟
Ħ��<[GYI�Y]Oh�YQXI�dg]GkEjQ][�G<j<�䘟
.IfkQgIZI[jh
¦�7P<j�h�qg][O���䘟
æ�7P<j�qQYY�O]�qg][O��䘟
Ħ��<[GYI�Y]Oh�YQXI�dg]GkEjQ][�G<j<�䘟
7P<j�Qh�<�Y]O�
�]q�j]�P<[GYI�Y]Oh�
�7<sh�]N�/I[GQ[O� ]Oh�j]�/]Yg
Y]OOIg
]Ohj<hP
NQYIh
/EPIZ<YIhh
�EG�h]Yg����Ir<ZdYI��Zp�h]Yg�h]Yg�D<X
�Ed��.�Ir<ZdYI�hEPIZ<YIhh�h]Yg���
�kj]Z<jQE����OI[Ig<jQ][
h]YgE][NQO�rZY
ßkdG<jI.IfkIhj+g]EIhh]g P<Q[�[<ZIÞ±<GG�k[X[]q[�NQIYGh�j]�jPI�hEPIZ<±à�������««��
ßdg]EIhh]g�EY<hhÞ±h]Yg�11��1dG<jI+g]EIhh]g�<Ej]gs±à���ßhjg�[<ZIÞ±NQIYG"<ZI±àQGß�hjgà�ß�dg]EIhh]gàßdg]EIhh]g�EY<hhÞ±h]Yg� ]O1dG<jI+g]EIhh]g�<Ej]gs±�àßdg]EIhh]g�EY<hhÞ±h]Yg�.k[1dG<jI+g]EIhh]g�<Ej]gs±�à
ß�kdG<jI.IfkIhj+g]EIhh]g P<Q[à
KWWS���VROU�SO�HQ������������DXWRPDWLFDOO\�JHQHUDWH�GRFXPHQW�LGHQWLILHUV�VROU���[�
Y]OOIg
�GIp�Y]O
ZZWh][d<ghI]Zdg]O�Û�hEgQdj
�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg
Ú�Y]OOIg�°³EII��©±PIYY]±��±q]gYG±ª°
ghshY]O�E][N
Z]GkYI¥Y]<GÞ±QZkrh]EX±¦�¾�pIghQ][�ÈÛ
�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg
���
Z]GkYI¥Y]<GÞ±ZZWh][d<ghI±¦
<EjQ][¥jsdIÞ±ZZWh][d<ghI±¦
�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg
���jIZdY<jI¥[<ZIÞ± ��±
����� jsdIÞ±YQhj±¦�©
������� dg]dIgjs¥[<ZIÞ±¹�<YY�Wh][±¦
������� E][hj<[j¥p<YkIÞ±�[±¦
����� ª
�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg
���<EjQ][¥jsdIÞ±ZZWh][d<ghI±¦jIZdY<jI¥[<ZIÞ± ��±«Z]GkYI¥Y]<GÞ±]Zdg]O±¦
QN�¹d<ghIhkEEIhh�ÞÞ�±$�±�jPI[�<EjQ][¥jsdIÞ±]Zdg]O±
��� DQ[<gsÞ±�]dj�Wh][�j]�h]Yg�ds±
��� jIZdY<jIÞ± ��±¦
�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg
QZd]gj�Wh][��dsh]Yg��hsh
h]Yg�Þ�dsh]Yg�/]Yg¥°Pjjd���Y]E<YP]hj�ÉÊÉÄ�h]Yg�°¦
qPQYI�0gkI�
��YQ[I�Þ�hsh�hjGQ[�gI<GYQ[I¥¦
��G]E�Þ�Wh][�Y]<Gh¥YQ[I¦
��h]Yg�<GG¥§G]E¨¦
�pg]
!]gdPYQ[I/]Yg�/Q[X
�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg
Pjjdh���OQjPkD�E]Z�ZdIgEs�NYkZI�Y]OÅW�Ir<ZdYI
NYkZI�E][N<OI[j�h]kgEIh�Þ�<pg]/gE
<OI[j�h]kgEIh�<pg]/gE�jsdI�Þ�<pg]
<OI[j�h]kgEIh�<pg]/gE�DQ[G�Þ�Á�Á�Á�Á
<OI[j�h]kgEIh�<pg]/gE�d]gj�Þ�ÅÂÅÂÅ
�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg
NYkZI�E][N
<OI[j�EP<[[IYh�Þ�h]Yg!IZ]gs P<[[IY
<OI[j�EP<[[IYh�h]Yg!IZ]gs P<[[IY�jsdI�Þ�ZIZ]gs
<OI[j�h]kgEIh�<pg]/gE�EP<[[IYh�Þ�h]Yg!IZ]gs P<[[IY
�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg
NYkZI�E][N
<OI[j�hQ[Xh�Þ�h]Yg/Q[X
<OI[j�hQ[Xh�h]Yg/Q[X�jsdI�Þ�]gO�<d<EPI�NYkZI�hQ[X�h]Yg�Z]gdPYQ[I�!]gdPYQ[I/]Yg/Q[X
<OI[j�hQ[Xh�h]Yg/Q[X�Z]gdPYQ[I�QYI�Þ�E][N�Z]gdPYQ[I�E][N
<OI[j�hQ[Xh�h]Yg/Q[X�EP<[[IY�Þ�h]Yg!IZ]gs P<[[IY
�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg
Z]gdPYQ[I�E][N����E]ZZ<[Gh���§
�� ©�gI<G Q[I�©�EP<ghIj���10��É�ªª
�� ©�Og]X�©
������ GQEjQ][<gs�QYIh���§E][N�Og]X�d<jjIg[h¨
������ IrdgIhhQ][h���©
�������� ZIhh<OI���±±±Ú©�"0�dQGª�Ú©��0��ZIhh<OIª±±±
���
KWWSV���JLWKXE�FRP�FORXGHUD�VHDUFK�WUHH�PDVWHU�VDPSOHV�VROU�QUW�JURN�GLFWLRQDULHV
�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg
Z]gdPYQ[I�E][N
/$ .¢ $ �0$.���©��E]YYIEjQ][���E]YYIEjQ][Â��¾vX�]hj���±ÂÃÈ�Á�Á�Â�ÃÂɱ��h]Yg1gY���±Pjjd���Y]E<YP]hj�ÉÊÉÄ�h]Yg�±ª����E]ZZ<[Gh���§
���
����©�Y]<G/]Yg�©
������ h]Yg ]E<j]g���¹©/$ .¢ $ �0$.ª
���
NYkI[j�Y]OOIg NYkI[j�dYkOQ[�h]Yg
NYkI[j�Y]OOIg��à�NYkI[jG��à�NYkI[j�dYkOQ[�h]Yg
Ú�dQd�Q[hj<YY�NYkI[j�Y]OOIg
Ng]Z�NYkI[j�QZd]gj�hI[GIg�IpI[j
hI[GIg�hIjkd¥°h]Yg�jIhj°¦
IpI[j��pI[j¥°N]gq<gG°��©°PIYY]°��°q]gYG°ª¦
NYkI[j�Y]OOIg��à�NYkI[jG��à�NYkI[j�dYkOQ[�h]Yg
ßh]kgEIà
��jsdI�N]gq<gG
ß�h]kgEIà
ßZ<jEP�h]Yg�««à
��jsdI�h]Yg
��P]hj�Y]E<YP]hj
��d]gj�ÉÊÉÄ
��E]gI�E]YYIEjQ][����������������������������� �
ß�Z<jEPà
NYkI[j�Y]OOIg��à�NYkI[jG��à�NYkI[j�dYkOQ[�h]Yg
Ú�OIZ�Q[hj<YY�NYkI[j�dYkOQ[�h]Yg
G]E�Þ�/]Yg���]EkZI[j�[Iq¥�PIYY]�Þà�gIE]gG§±PIYY]±¨¦
KWWSV���JLWKXE�FRP�EWLJLW�IOXHQW�SOXJLQ�VROU
]kj¢h]Yg�gD
NQYI�Q[dkj h]Yg¢Pjjd�]kjdkj
]Ohj<hPNQYI
Og]X�NQYjIg
Y]Ohj<hP�E][N�
Q[dkj�©��NQYI�©�d<jP�Þà�±�jZd�jIhjY]O±�ªª
NQYI�Q[dkj��à�Og]X�NQYjIg��à�h]Yg¢Pjjd�]kjdkj
Ú�IEP]�°Ã�q]gYG°�àà��jZd�jIhjY]O
Y]Ohj<hP�E][N�
NQYjIg�©��Og]X�©����Z<jEP�Þà�§±ZIhh<OI±��±Ú©"1!��.�dQGª�Ú©�.���9��0��PIYY]ª±¨��ªª
NQYI�Q[dkj��à�Og]X�NQYjIg��à�h]Yg¢Pjjd�]kjdkj
©±dQG±��±Ã±��±PIYY]±�±q]gYG±ª
Y]Ohj<hP�E][N�
]kjdkj�©��h]Yg¢Pjjd�©����¾�Z<hjIg�]g�pÂ�Ã�ÄÛ����h]Yg¢kgY�Þà�±Pjjd���Y]E<YP]hj�ÉÊÉÄ�h]Yg±��ªª
NQYI�Q[dkj��à�Og]X�NQYjIg��à�h]Yg¢Pjjd�]kjdkj
�<hj�<[G� Y]kG
��j��IdI[Gh�
KWWS���ZZZ�ELJVN\WHFK�FRP�ZS�FRQWHQW�XSORDGV���������JXDJH�SQJ
Y]<G�jIhj Z][Qj]g��/+!
ÃÁÚ�]NN�� .ÃÁÂÄ/+!ÃÁ
�àààà�/Q[OYI� ]gI��¾�]N�G]Eh�kdG<jI
KWWS���VWDWLF�PHPULVH�FRP�V��DPD]RQDZV�FRP�XSORDGV�EORJ�SLFWXUHV�6LPSVRQVB8SGDWHV�EPS
�àààà�/Q[OYI� ]gI�� ]ZZQjh
KWWS���FDFKH�GHVNWRSQH[XV�FRP�WKXPEQDLOV������ELJWKXPEQDLO�MSJKWWS���ZZZ�PXVLFIHVWLYDOMXQNLHV�FRP�ZS�FRQWHQW�XSORDGV���������+DUG/RJR�SQJ
ß<kj]/]Nj ]ZZQjà��ßZ<r0QZIà���
ß<kj] ]ZZQjà��ß]dI[/I<gEPIgàN<YhI��ßZ<r0QZIà���
ßg<Z�kNNIg/QvI!�à���
�àààà�/Q[OYI� ]gI��/QvI�<[G�!IgOIh
KWWS���VZHHWFOLSDUW�FRP�PXOWLVLWH�VZHHWFOLSDUW�ILOHV�VFLVVRUVBEOXHBVLOYHU�SQJKWWS���PHUJHZRUGV�FRP�JI[�ORJR�ELJ�SQJ
]ZQj"]gZhÞ±jgkI±]ZQj0IgZ�gIf�[G+]hQjQ][hÞ±jgkI± ßZIgOI�<Ej]gà��
�àààà�/Q[OYI� ]gI�� <EPIh
KWWS���YHFWRU�PDJ]�FRP�ZS�FRQWHQW�XSORDGV���������GLDPRQG�FOLS�DUW��SQJKWWS���ZZZ�FONHU�FRP�FOLSDUWV���I�������������������������6DUD6DUDB,FHBFXEHB��VYJ�PHG�SQJ
KWWS���FOLSDUWLVW�LQIR�566�RSHQFOLSDUW�RUJ������0D\����0RQGD\�PLJUDWLQJBSHQJXLQBSHQJXLQPLJUDWLQJ����S[�SQJ
��� ßNQIYG6<YkI <EPI������������������������ hQvIÞ±���±�������������������� <kj]q<gZ ]k[jÞ±Á±
�������������������������������G]E6<YkIhÞ±jgkI±
N<EIjh
EP<[OQ[O�G<j<j]�h]gj²N<EIj
/]Yg Y]kG��;]]�IIdIg
DQ[�vX/IgpIg�hP�hj<gj
$.
W<p<���vX.k[�«��W<g�hj<gj�W<gKWWS���ZZZ�FONHU�FRP�FOLSDUWV�F�D���G��������������������5RDULQJ���7LJHU�VYJ�KL�SQJ
KWWS���IF���GHYLDQWDUW�QHW�IV���I������������D�SLJJ\BEDFNBULGHVBDUHBWKHBEHVWBULGHVBBE\B\LSSHG�G��E�VK�SQJ
/]Yg Y]kG��;]]�IIdIg
vXEYQ�hP��EZG�kdE][NQO�����������������vXP]hj�/�.6�.�ÃÂÉÂ�����������������E][NGQg�h]Yg�E]YYIEjQ][Â�E][N������������������E][N[<ZI�hj<gj
��D]]jhjg<d¢E][NGQgÞh]Yg�E]YYIEjQ][Â�E][N���E]YYIEjQ][�E][NQO"<ZIÞhj<gj
KWWS���ZZZ�FONHU�FRP�FOLSDUWV�F�D���G��������������������5RDULQJ���7LJHU�VYJ�KL�SQJKWWS���IF���GHYLDQWDUW�QHW�IV���I������������D�SLJJ\BEDFNBULGHVBDUHBWKHBEHVWBULGHVBBE\B\LSSHG�G��E�VK�SQJ
/]Yg Y]kG��/j<gj�"]GIh
W<p<���vX�]hjÞ/�.6�.�ÃÂÉÂ��W<g�hj<gj�W<g
0QZIG� ]YYIEjQ][h
ÁÅ"]p
ÁÆ"]p
ÁÇ�"]p
ÁÈ"]p
hI<gEP�Y<jIhj
hI<gEP�<YY
Q[GIr
]djQZQvI
]YYIEjQ][h��+�
ÁÆ"]p
ÁÇ"]p
ÁÈ�"]p
ÁÉ"]p
<EjQ][Þ .��0�²[<ZIÞÁÉ"]p²[kZ/P<gGhÞÅ
<EjQ][Þ�� �0�²[<ZIÞÁÆ"]p
�YQ<hIh��$djQZQvI
ÁÆ"]p
ÁÇ"]p
ÁÈ�"]p
ÁÉ"]p
<EjQ][Þ .��0�� ��/²[<ZIÞ� ²E]YYIEjQ][ÞÁÇ"]p�ÁÈ"]p�ÁÉ"]p
<EjQ][Þ .��0�� ��/²[<ZIÞ �0�/0²E]YYIEjQ][ÞÁÉ"]pÁÈ"]p�kdG<jI�]djQZQvIÞjgkI
Y]Oh�Þdg]GkEjQ][�
G<j<
Y]Oh�Þdg]GkEjQ][�
G<j<
]Ohj<hP
Y]Oh�Þdg]GkEjQ][�
G<j<
]Ohj<hP
G]Eh�kdG<jIE]ZZQjh
ZIgOI�<Ej]g
]ZQj«G]E6<YkIh
E<EPIh
Y]Oh�Þdg]GkEjQ][�
G<j<
]Ohj<hP
G]Eh�kdG<jIE]ZZQjh
ZIgOI�<Ej]g
]ZQj«G]E6<YkIh
E<EPIh
Y]Oh�Þdg]GkEjQ][�
G<j<
]Ohj<hP
G]Eh�kdG<jIE]ZZQjh
ZIgOI�<Ej]g
]ZQj«G]E6<YkIh
E<EPIh
jQZI
]YYIEjQ][h��+�<YQ<hIh
]djQZQvI
7I�gI�PQgQ[O�
hIZ<jIrj�E]Z�<D]kj�W]Dh
0P<[X�s]k�
UDGX�JKHRUJKH#VHPDWH[W�FRP³g<GkÁOPI]gOPI��������������������������³hIZ<jIrj
�[G�³�]kg�D]]jP��¦