using solr to search and analyze logs

60
1hQ[O /]Yg j] /I<gEP <[G [<YsvI ]Oh .<Gk PI]gOPI ³g<GkÁOPI]gOPI ³hIZ<jIrj

Upload: lucenerevolution

Post on 11-May-2015

969 views

Category:

Technology


6 download

DESCRIPTION

Many of us tend to hate or simply ignore logs, and rightfully so: they’re typically hard to find, difficult to handle, and are cryptic to the human eye. But can we make logs more valuable and more usable if we index them in Solr, so we can search and run real-time statistics on them? Indeed we can, and in this session you’ll learn how to make that happen. In the first part of the session we’ll explain why centralized logging is important, what valuable information one can extract from logs, and we’ll introduce the leading tools from the logging ecosystems everyone should be aware of - from syslog and log4j to LogStash and Flume. In the second part we’ll teach you how to use these tools in tandem with Solr. We’ll show how to use Solr in a SolrCloud setup to index large volumes of logs continuously and efficiently. Then, we'll look at how to scale the Solr cluster as your data volume grows. Finally, we'll see how you can parse your unstructured logs and convert them to nicely structured Solr documents suitable for analytical queries.

TRANSCRIPT

Page 1: Using Solr to Search and Analyze Logs

1hQ[O�/]Yg�j]�/I<gEP�<[G�

�[<YsvI� ]Oh

.<Gk��PI]gOPI

³g<GkÁOPI]gOPI³hIZ<jIrj

Page 2: Using Solr to Search and Analyze Logs

�Y<hjQEhI<gEP��+�

hshY]OgIEIQpIg

]OhI[I

�QD<[<

hshY]OG

]Ohj<hP

Page 4: Using Solr to Search and Analyze Logs

7P<j�<D]kj �

Page 5: Using Solr to Search and Analyze Logs

GINQ[Q[O�<[G�P<[GYQ[O�Y]Oh�Q[�OI[Ig<Y

�hIjh�]N�j]]Yh�j]�hI[G�Y]Oh�j]

+IgN]gZ<[EI�jk[Q[O�<[G�/]Yg Y]kG

Page 6: Using Solr to Search and Analyze Logs

hshY]O

�INQ[Q[O�<[G��<[GYQ[O� ]Oh¥hj]gs�jQZI�¦

hshY]O

hshY]O

hshY]O

Page 7: Using Solr to Search and Analyze Logs

.IfkQgIZI[jh

¦�7P<j�h�qg][O�

KWWS���HGG\VXDLE�FRP�ZS�FRQWHQW�XSORDGV���������.H\ZRUG�LFRQ�SQJ

¥���������������N]g�GIDkOOQ[O¦

Page 8: Using Solr to Search and Analyze Logs

+g]DYIZ

Y]]]]]jh�]N�ZIhh<OIh�E]ZQ[O�Q[

KWWS���ZZZ�VFLHQFHVXUYLYDOEORJ�FRP�JHWWLQJ�SXEOLVKHG�XQILQLVKHG�PDQXVFULSWVB����

Page 9: Using Solr to Search and Analyze Logs

/]YpIG�qQjP�[]�Q[GIrQ[O

�10

Page 10: Using Solr to Search and Analyze Logs

�Y<hjQEhI<gEP

Page 11: Using Solr to Search and Analyze Logs

.IfkQgIZI[jh

¦�7P<j�h�qg][O���䘟

æ�7P<j�qQYY�O]�qg][O�

�������������¥hj<jh¦

Page 12: Using Solr to Search and Analyze Logs

+<ghQ[O�.<q� ]Oh

�10

ZQEXIs�Z]khI�ÂÁ

khIg QjIZ jQZI

hjQYY�hY]q N]gZ<j�EP<[OIh

Page 13: Using Solr to Search and Analyze Logs

+<ghQ[O�.<q� ]Oh

�10

ZQEXIs�Z]khI�Á�ÂÁ

<GG�Igg]g�E]GI

hjQYY�hY]q N]gZ<j�EP<[OIh

Page 14: Using Solr to Search and Analyze Logs

�<EIjh�� ]OOQ[O�Q[��/$"

ÃÁÂÄ�ÂÂ�ÁÇ«�ZQEXIs�Z]khI

©��±G<jI±��±ÃÁÂÄ�ÂÂ�ÁDZ���±ZIhh<OI±��±ZQEXIs�Z]khI±ª

Page 15: Using Solr to Search and Analyze Logs

�<EIjh�� ]OOQ[O�Q[��/$"

ÃÁÂÄ�ÂÂ�ÁÇ«�³EII�©±khIg±��±ZQEXIs±ª

©��±G<jI±��±ÃÁÂÄ�ÂÂ�ÁDZ���±khIg±��±ZQEXIs±ª

ÃÁÂÄ�ÂÂ�ÁÇ«�ZQEXIs�Z]khI

©��±G<jI±��±ÃÁÂÄ�ÂÂ�ÁDZ���±ZIhh<OI±��±ZQEXIs�Z]khI±ª

Page 16: Using Solr to Search and Analyze Logs

.IfkQgIZI[jh

¦�7P<j�h�qg][O���䘟

æ�7P<j�qQYY�O]�qg][O��䘟

Ħ��<[GYI�Y]Oh�YQXI�dg]GkEjQ][�G<j<�䘟

Page 17: Using Solr to Search and Analyze Logs

.IfkQgIZI[jh

¦�7P<j�h�qg][O���䘟

æ�7P<j�qQYY�O]�qg][O��䘟

Ħ��<[GYI�Y]Oh�YQXI�dg]GkEjQ][�G<j<�䘟

7P<j�Qh�<�Y]O�

�]q�j]�P<[GYI�Y]Oh�

Page 18: Using Solr to Search and Analyze Logs

�7<sh�]N�/I[GQ[O� ]Oh�j]�/]Yg

Y]OOIg

]Ohj<hP

NQYIh

Page 19: Using Solr to Search and Analyze Logs

/EPIZ<YIhh

�EG�h]Yg����Ir<ZdYI��Zp�h]Yg�h]Yg�D<X

�Ed��.�Ir<ZdYI�hEPIZ<YIhh�h]Yg���

Page 20: Using Solr to Search and Analyze Logs

�kj]Z<jQE����OI[Ig<jQ][

h]YgE][NQO�rZY

ßkdG<jI.IfkIhj+g]EIhh]g P<Q[�[<ZIÞ±<GG�k[X[]q[�NQIYGh�j]�jPI�hEPIZ<±à�������««��

ßdg]EIhh]g�EY<hhÞ±h]Yg�11��1dG<jI+g]EIhh]g�<Ej]gs±à���ßhjg�[<ZIÞ±NQIYG"<ZI±àQGß�hjgà�ß�dg]EIhh]gàßdg]EIhh]g�EY<hhÞ±h]Yg� ]O1dG<jI+g]EIhh]g�<Ej]gs±�àßdg]EIhh]g�EY<hhÞ±h]Yg�.k[1dG<jI+g]EIhh]g�<Ej]gs±�à

ß�kdG<jI.IfkIhj+g]EIhh]g P<Q[à

KWWS���VROU�SO�HQ������������DXWRPDWLFDOO\�JHQHUDWH�GRFXPHQW�LGHQWLILHUV�VROU���[�

Page 21: Using Solr to Search and Analyze Logs

Y]OOIg

�GIp�Y]O

ZZWh][d<ghI]Zdg]O�Û�hEgQdj

Page 22: Using Solr to Search and Analyze Logs

�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg

Ú�Y]OOIg�°³EII��©±PIYY]±��±q]gYG±ª°

ghshY]O�E][N

Z]GkYI¥Y]<GÞ±QZkrh]EX±¦�¾�pIghQ][�ÈÛ

Page 23: Using Solr to Search and Analyze Logs

�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg

���

Z]GkYI¥Y]<GÞ±ZZWh][d<ghI±¦

<EjQ][¥jsdIÞ±ZZWh][d<ghI±¦

Page 24: Using Solr to Search and Analyze Logs

�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg

���jIZdY<jI¥[<ZIÞ± ��±

����� jsdIÞ±YQhj±¦�©

������� dg]dIgjs¥[<ZIÞ±¹�<YY�Wh][±¦

������� E][hj<[j¥p<YkIÞ±�[±¦

����� ª

Page 25: Using Solr to Search and Analyze Logs

�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg

���<EjQ][¥jsdIÞ±ZZWh][d<ghI±¦jIZdY<jI¥[<ZIÞ± ��±«Z]GkYI¥Y]<GÞ±]Zdg]O±¦

QN�¹d<ghIhkEEIhh�ÞÞ�±$�±�jPI[�<EjQ][¥jsdIÞ±]Zdg]O±

��� DQ[<gsÞ±�]dj�Wh][�j]�h]Yg�ds±

��� jIZdY<jIÞ± ��±¦

Page 26: Using Solr to Search and Analyze Logs

�GIp�Y]O��à�d<ghI��à�N]gZ<j��à�hI[G�j]�/]Yg

QZd]gj�Wh][��dsh]Yg��hsh

h]Yg�Þ�dsh]Yg�/]Yg¥°Pjjd���Y]E<YP]hj�ÉÊÉÄ�h]Yg�°¦

qPQYI�0gkI�

��YQ[I�Þ�hsh�hjGQ[�gI<GYQ[I¥¦

��G]E�Þ�Wh][�Y]<Gh¥YQ[I¦

��h]Yg�<GG¥§G]E¨¦

Page 27: Using Solr to Search and Analyze Logs

�pg]

!]gdPYQ[I/]Yg�/Q[X

Page 28: Using Solr to Search and Analyze Logs

�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg

Pjjdh���OQjPkD�E]Z�ZdIgEs�NYkZI�Y]OÅW�Ir<ZdYI

NYkZI�E][N<OI[j�h]kgEIh�Þ�<pg]/gE

<OI[j�h]kgEIh�<pg]/gE�jsdI�Þ�<pg]

<OI[j�h]kgEIh�<pg]/gE�DQ[G�Þ�Á�Á�Á�Á

<OI[j�h]kgEIh�<pg]/gE�d]gj�Þ�ÅÂÅÂÅ

Page 29: Using Solr to Search and Analyze Logs

�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg

NYkZI�E][N

<OI[j�EP<[[IYh�Þ�h]Yg!IZ]gs P<[[IY

<OI[j�EP<[[IYh�h]Yg!IZ]gs P<[[IY�jsdI�Þ�ZIZ]gs

<OI[j�h]kgEIh�<pg]/gE�EP<[[IYh�Þ�h]Yg!IZ]gs P<[[IY

Page 30: Using Solr to Search and Analyze Logs

�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg

NYkZI�E][N

<OI[j�hQ[Xh�Þ�h]Yg/Q[X

<OI[j�hQ[Xh�h]Yg/Q[X�jsdI�Þ�]gO�<d<EPI�NYkZI�hQ[X�h]Yg�Z]gdPYQ[I�!]gdPYQ[I/]Yg/Q[X

<OI[j�hQ[Xh�h]Yg/Q[X�Z]gdPYQ[I�QYI�Þ�E][N�Z]gdPYQ[I�E][N

<OI[j�hQ[Xh�h]Yg/Q[X�EP<[[IY�Þ�h]Yg!IZ]gs P<[[IY

Page 31: Using Solr to Search and Analyze Logs

�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg

Z]gdPYQ[I�E][N����E]ZZ<[Gh���§

�� ©�gI<G Q[I�©�EP<ghIj���10��É�ªª

�� ©�Og]X�©

������ GQEjQ][<gs�QYIh���§E][N�Og]X�d<jjIg[h¨

������ IrdgIhhQ][h���©

�������� ZIhh<OI���±±±Ú©�"0�dQGª�Ú©��0��ZIhh<OIª±±±

���

KWWSV���JLWKXE�FRP�FORXGHUD�VHDUFK�WUHH�PDVWHU�VDPSOHV�VROU�QUW�JURN�GLFWLRQDULHV

Page 32: Using Solr to Search and Analyze Logs

�pg]��à�DkNNIg��à�d<ghI��à�hI[G�j]�/]Yg

Z]gdPYQ[I�E][N

/$ .¢ $ �0$.���©��E]YYIEjQ][���E]YYIEjQ][Â��¾vX�]hj���±ÂÃÈ�Á�Á�Â�ÃÂɱ��h]Yg1gY���±Pjjd���Y]E<YP]hj�ÉÊÉÄ�h]Yg�±ª����E]ZZ<[Gh���§

���

����©�Y]<G/]Yg�©

������ h]Yg ]E<j]g���¹©/$ .¢ $ �0$.ª

���

Page 33: Using Solr to Search and Analyze Logs

NYkI[j�Y]OOIg NYkI[j�dYkOQ[�h]Yg

Page 34: Using Solr to Search and Analyze Logs

NYkI[j�Y]OOIg��à�NYkI[jG��à�NYkI[j�dYkOQ[�h]Yg

Ú�dQd�Q[hj<YY�NYkI[j�Y]OOIg

Ng]Z�NYkI[j�QZd]gj�hI[GIg�IpI[j

hI[GIg�hIjkd¥°h]Yg�jIhj°¦

IpI[j��pI[j¥°N]gq<gG°��©°PIYY]°��°q]gYG°ª¦

Page 35: Using Solr to Search and Analyze Logs

NYkI[j�Y]OOIg��à�NYkI[jG��à�NYkI[j�dYkOQ[�h]Yg

ßh]kgEIà

��jsdI�N]gq<gG

ß�h]kgEIà

ßZ<jEP�h]Yg�««à

��jsdI�h]Yg

��P]hj�Y]E<YP]hj

��d]gj�ÉÊÉÄ

��E]gI�E]YYIEjQ][����������������������������� �

ß�Z<jEPà

Page 36: Using Solr to Search and Analyze Logs

NYkI[j�Y]OOIg��à�NYkI[jG��à�NYkI[j�dYkOQ[�h]Yg

Ú�OIZ�Q[hj<YY�NYkI[j�dYkOQ[�h]Yg

G]E�Þ�/]Yg���]EkZI[j�[Iq¥�PIYY]�Þà�gIE]gG§±PIYY]±¨¦

KWWSV���JLWKXE�FRP�EWLJLW�IOXHQW�SOXJLQ�VROU

]kj¢h]Yg�gD

Page 37: Using Solr to Search and Analyze Logs

NQYI�Q[dkj h]Yg¢Pjjd�]kjdkj

]Ohj<hPNQYI

Og]X�NQYjIg

Page 38: Using Solr to Search and Analyze Logs

Y]Ohj<hP�E][N�

Q[dkj�©��NQYI�©�d<jP�Þà�±�jZd�jIhjY]O±�ªª

NQYI�Q[dkj��à�Og]X�NQYjIg��à�h]Yg¢Pjjd�]kjdkj

Ú�IEP]�°Ã�q]gYG°�àà��jZd�jIhjY]O

Page 39: Using Solr to Search and Analyze Logs

Y]Ohj<hP�E][N�

NQYjIg�©��Og]X�©����Z<jEP�Þà�§±ZIhh<OI±��±Ú©"1!��.�dQGª�Ú©�.���9��0��PIYY]ª±¨��ªª

NQYI�Q[dkj��à�Og]X�NQYjIg��à�h]Yg¢Pjjd�]kjdkj

©±dQG±��±Ã±��±PIYY]±�±q]gYG±ª

Page 40: Using Solr to Search and Analyze Logs

Y]Ohj<hP�E][N�

]kjdkj�©��h]Yg¢Pjjd�©����¾�Z<hjIg�]g�pÂ�Ã�ÄÛ����h]Yg¢kgY�Þà�±Pjjd���Y]E<YP]hj�ÉÊÉÄ�h]Yg±��ªª

NQYI�Q[dkj��à�Og]X�NQYjIg��à�h]Yg¢Pjjd�]kjdkj

Page 41: Using Solr to Search and Analyze Logs

�<hj�<[G� Y]kG

Page 42: Using Solr to Search and Analyze Logs

��j��IdI[Gh�

KWWS���ZZZ�ELJVN\WHFK�FRP�ZS�FRQWHQW�XSORDGV���������JXDJH�SQJ

Y]<G�jIhj Z][Qj]g��/+!

ÃÁÚ�]NN�� .ÃÁÂÄ/+!ÃÁ

Page 43: Using Solr to Search and Analyze Logs

�àààà�/Q[OYI� ]gI��¾�]N�G]Eh�kdG<jI

KWWS���VWDWLF�PHPULVH�FRP�V��DPD]RQDZV�FRP�XSORDGV�EORJ�SLFWXUHV�6LPSVRQVB8SGDWHV�EPS

Page 44: Using Solr to Search and Analyze Logs

�àààà�/Q[OYI� ]gI�� ]ZZQjh

KWWS���FDFKH�GHVNWRSQH[XV�FRP�WKXPEQDLOV������ELJWKXPEQDLO�MSJKWWS���ZZZ�PXVLFIHVWLYDOMXQNLHV�FRP�ZS�FRQWHQW�XSORDGV���������+DUG/RJR�SQJ

ß<kj]/]Nj ]ZZQjà��ßZ<r0QZIà���

ß<kj] ]ZZQjà��ß]dI[/I<gEPIgàN<YhI��ßZ<r0QZIà���

ßg<Z�kNNIg/QvI!�à���

Page 45: Using Solr to Search and Analyze Logs

�àààà�/Q[OYI� ]gI��/QvI�<[G�!IgOIh

KWWS���VZHHWFOLSDUW�FRP�PXOWLVLWH�VZHHWFOLSDUW�ILOHV�VFLVVRUVBEOXHBVLOYHU�SQJKWWS���PHUJHZRUGV�FRP�JI[�ORJR�ELJ�SQJ

]ZQj"]gZhÞ±jgkI±]ZQj0IgZ�gIf�[G+]hQjQ][hÞ±jgkI± ßZIgOI�<Ej]gà��

Page 46: Using Solr to Search and Analyze Logs

�àààà�/Q[OYI� ]gI�� <EPIh

KWWS���YHFWRU�PDJ]�FRP�ZS�FRQWHQW�XSORDGV���������GLDPRQG�FOLS�DUW��SQJKWWS���ZZZ�FONHU�FRP�FOLSDUWV���I�������������������������6DUD6DUDB,FHBFXEHB��VYJ�PHG�SQJ

KWWS���FOLSDUWLVW�LQIR�566�RSHQFOLSDUW�RUJ������0D\����0RQGD\�PLJUDWLQJBSHQJXLQBSHQJXLQPLJUDWLQJ����S[�SQJ

��� ßNQIYG6<YkI <EPI������������������������ hQvIÞ±���±�������������������� <kj]q<gZ ]k[jÞ±Á±

�������������������������������G]E6<YkIhÞ±jgkI±

N<EIjh

EP<[OQ[O�G<j<j]�h]gj²N<EIj

Page 47: Using Solr to Search and Analyze Logs

/]Yg Y]kG��;]]�IIdIg

DQ[�vX/IgpIg�hP�hj<gj

$.

W<p<���vX.k[�«��W<g�hj<gj�W<gKWWS���ZZZ�FONHU�FRP�FOLSDUWV�F�D���G��������������������5RDULQJ���7LJHU�VYJ�KL�SQJ

KWWS���IF���GHYLDQWDUW�QHW�IV���I������������D�SLJJ\BEDFNBULGHVBDUHBWKHBEHVWBULGHVBBE\B\LSSHG�G��E�VK�SQJ

Page 48: Using Solr to Search and Analyze Logs

/]Yg Y]kG��;]]�IIdIg

vXEYQ�hP��EZG�kdE][NQO�����������������vXP]hj�/�.6�.�ÃÂÉÂ�����������������E][NGQg�h]Yg�E]YYIEjQ][Â�E][N������������������E][N[<ZI�hj<gj

��D]]jhjg<d¢E][NGQgÞh]Yg�E]YYIEjQ][Â�E][N���E]YYIEjQ][�E][NQO"<ZIÞhj<gj

KWWS���ZZZ�FONHU�FRP�FOLSDUWV�F�D���G��������������������5RDULQJ���7LJHU�VYJ�KL�SQJKWWS���IF���GHYLDQWDUW�QHW�IV���I������������D�SLJJ\BEDFNBULGHVBDUHBWKHBEHVWBULGHVBBE\B\LSSHG�G��E�VK�SQJ

Page 49: Using Solr to Search and Analyze Logs

/]Yg Y]kG��/j<gj�"]GIh

W<p<���vX�]hjÞ/�.6�.�ÃÂÉÂ��W<g�hj<gj�W<g

Page 50: Using Solr to Search and Analyze Logs

0QZIG� ]YYIEjQ][h

ÁÅ"]p

ÁÆ"]p

ÁÇ�"]p

ÁÈ"]p

hI<gEP�Y<jIhj

hI<gEP�<YY

Q[GIr

]djQZQvI

Page 51: Using Solr to Search and Analyze Logs

]YYIEjQ][h��+�

ÁÆ"]p

ÁÇ"]p

ÁÈ�"]p

ÁÉ"]p

<EjQ][Þ .��0�²[<ZIÞÁÉ"]p²[kZ/P<gGhÞÅ

<EjQ][Þ�� �0�²[<ZIÞÁÆ"]p

Page 52: Using Solr to Search and Analyze Logs

�YQ<hIh��$djQZQvI

ÁÆ"]p

ÁÇ"]p

ÁÈ�"]p

ÁÉ"]p

<EjQ][Þ .��0�� ��/²[<ZIÞ� ²E]YYIEjQ][ÞÁÇ"]p�ÁÈ"]p�ÁÉ"]p

<EjQ][Þ .��0�� ��/²[<ZIÞ �0�/0²E]YYIEjQ][ÞÁÉ"]pÁÈ"]p�kdG<jI�]djQZQvIÞjgkI

Page 53: Using Solr to Search and Analyze Logs
Page 54: Using Solr to Search and Analyze Logs

Y]Oh�Þdg]GkEjQ][�

G<j<

Page 55: Using Solr to Search and Analyze Logs

Y]Oh�Þdg]GkEjQ][�

G<j<

]Ohj<hP

Page 56: Using Solr to Search and Analyze Logs

Y]Oh�Þdg]GkEjQ][�

G<j<

]Ohj<hP

G]Eh�kdG<jIE]ZZQjh

ZIgOI�<Ej]g

]ZQj«G]E6<YkIh

E<EPIh

Page 57: Using Solr to Search and Analyze Logs

Y]Oh�Þdg]GkEjQ][�

G<j<

]Ohj<hP

G]Eh�kdG<jIE]ZZQjh

ZIgOI�<Ej]g

]ZQj«G]E6<YkIh

E<EPIh

Page 58: Using Solr to Search and Analyze Logs

Y]Oh�Þdg]GkEjQ][�

G<j<

]Ohj<hP

G]Eh�kdG<jIE]ZZQjh

ZIgOI�<Ej]g

]ZQj«G]E6<YkIh

E<EPIh

jQZI

]YYIEjQ][h��+�<YQ<hIh

]djQZQvI

Page 60: Using Solr to Search and Analyze Logs

0P<[X�s]k�

UDGX�JKHRUJKH#VHPDWH[W�FRP³g<GkÁOPI]gOPI��������������������������³hIZ<jIrj

�[G�³�]kg�D]]jP��¦