meetup070416 presentations
Post on 11-Apr-2017
222 Views
Preview:
TRANSCRIPT
3
ElasticSearch - Know Your Data
Elastic Stack
Polarisign 2014 3
Kibana
ElasticSearch
Logstash Beats
Plu
gin
s
Data Visualization
Store, Index, Search, Analyse
Data Ingestion
4
ElasticSearch - Know Your Data
ElasticSearch – Store, Index, Search and Analyze
Polarisign 2014 4
Distributed and Scalable
﹥ Design to scale-out: improved resiliency﹥ High Availability﹥ Multi tenant
Search and Analytics
﹥ Real-time (near)﹥ Full text search﹥ Multi language support﹥ Aggregations﹥ Geospacial Info support
Developing Key Features
﹥ Schemaless (structure and unstructured data)﹥ Document oriented (JSON)﹥ RESTful API﹥ Client Libraries (Java, .Net, PHP, etc)﹥ Build on top of Apache Lucene
5
ElasticSearch - Know Your Data
ElasticSearch – Scale-Out and High-Availability
Polarisign 2014 5
Elastic Cluster
Node 1
Shard 1 Shard 2
Shard 3 Shard 4
Elastic Cluster
Node 1
Shard 1 Shard 2
Shard 3 Shard 4
Node 2
Shard 1 Shard 2
Shard 3 Shard 4
Add one node
• 1 index• Nr of shards: 4• Nr of Replicas: 1
6
ElasticSearch - Know Your Data
Logstash – Data Ingest
Polarisign 2014 6
Collect, Enrich and Transport Data
﹥ Centralize data processing﹥ Normalize/Format distinct data types﹥ Easily extensible﹥ 200+ plugins
Data Pipeline
Input Filter(s) Output
• elasticsearch• file• syslog• rabbitmq
• csv• geoip• elasticsearch• mutate
• file• mongodb• elasticsearch• rabbitmq
7
ElasticSearch - Know Your Data
Beats – Data Ingest
Polarisign 2014 7
Collect, Parse and Ship Data
﹥ Lightweight data shippers﹥ Forward host-based operational metrics﹥ Single purpose﹥ Libbeat, framework to build new Beats
Operational Data Examples
﹥ Wire Data => Packetbeat (multiple decoders available, like HTTP, MySql)﹥ System Stats => Topbeat﹥ Logs => Filebeat, Winlogbeat
8
ElasticSearch - Know Your Data
Kibana – Data Visualization
Polarisign 2014 8
Search and Analytics
﹥ Web based interface for data visualization stored in ES﹥ Explore and analyze patterns in data﹥ Leverage the power of Elasticsearch analytics capabilities
Visualization
﹥ Supports multiple types, charts, maps, histograms﹥ Share and embed dashboards into operational dashaboards﹥ Supports custom visualizations and applications﹥ Plugins for cluster management and administration
9
ElasticSearch - Know Your Data
Elastic – Use Cases
Polarisign 2014 9
Search
﹥ Full text search (fast)﹥ Fuzzy text search﹥ Geospacial
Analytics
﹥ Explore your data﹥ Ask complex queries about your data﹥ Leveraged by ES aggregations feature﹥ Kibana!
Logging
﹥ Logstash or Beats => ElasticSearch => Kibana﹥ Centralize log store, analytics and visualization
10
ElasticSearch - Know Your Data
Elastic – Aggregations
Polarisign 2014 10
﹥ Summarize our data vs looking for particular documents
Sample Questions
﹥ Most popular items?﹥ Average shopping value for each day?﹥ Which stores sells more?﹥ Sky is the limit …
﹥ Quick and near real-time, just like search﹥ Powerful for reports and dashboards => No need for long run jobs
﹥ Can be combined with search/filter queries:﹥ Most popular items? => Most popular Electronic items?
11
ElasticSearch - Know Your Data
Elastic – Aggregations
Polarisign 2014 11
Concepts
﹥ Buckets: Collections of documents that meet a criterion﹥ Metrics: Statistics calculated on the documents in a bucket﹥ An aggregation is a combination of one or more buckets and zero or more metrics
SQL
SELECT COUNT(product)
FROM SALE
GROUP BY product
GET sales/_search{"size": 0, "aggs": {
"products": {"terms": {
"field": “product","size": 100
}}
}}
Elastic
12
ElasticSearch - Know Your Data
Elastic – Aggregations
Polarisign 2014 12
Concepts
﹥ Buckets: Collections of documents that meet a criterion﹥ Metrics: Statistics calculated on the documents in a bucket﹥ An aggregation is a combination of one or more buckets and zero or more metrics
SQL
SELECT COUNT(product)
FROM SALE
GROUP BY product
GET sales/_search{"size": 0, "aggs": {
"products": {"terms": {
"field": “product","size": 100
}}
}}
Elastic
13
ElasticSearch - Know Your Data
Elastic – Aggregations
Polarisign 2014 13
Types
﹥ Bucketing: builds buckets according to a criteria﹥ Histogram, Date Histogram, Geo Distance, Terms
﹥ Metrics: compute metrics over a set of documents﹥ Avg, Stats, Sum, Top Hits
﹥ Pipeline: aggregate the output of other aggregations and their associated metrics
Sub-Aggregation
﹥ Bucketing aggregations can have sub-aggregations﹥ No hard-limit of nested aggregation﹥ Sub-aggregations will execute within the context of parent buckets
20
POST /_reindex
{
"source": {
"index": "twitter"
},
"dest": {
"index": "new_twitter"
}
}
Reindex API
2.3
21
POST /twitter/_update_by_query
{
"script": {
"inline": "ctx._source.likes++"
},
"query": {
"term": {
"user": "kimchy"
}
}
}
Update by Query
2.3
30
GET /_logstash
GET /_logstash/${role}/config/current
POST /_logstash/${role}/config/_rollback
GET /_logstash/${role}/config/${version_number}
GET /_logstash/${role}/nodes/${node_name}
Management API5.x
31
What else in..
•Community Maintainers <3
• Dead letter queues•Centralized configuration
• Management UIs
Logstash
Source: Gray Arial 10pt
33
What to talk about
• Redis/Kafka output
• Filtering
• Metricbeat
• Tons of new community beats!!
Beats
Source: Gray Arial 10pt
35
Ability to customize colors, text,
numbers, labels, layouts, skins,
and visualizations.
1 All-new Visualization tools for
Graph and Time Series data.
2 Strong integration with Security,
Monitoring, and the rest of the
Elastic Stack
3
37
What to talk about
• space efficiency up 20% => more real estate
• export all the things!
• plugin all the things!
• kibana app generator
• status page
Kibana
Source: Gray Arial 10pt
45
What else in..
• Query Profiler
• Painless scripting engine•New relevancy scoring algo (BM-25)•Security Manager on by default
• Move from string field to keyword/text
• Java HTTP Client
• So many geo improvements!
Elasticsearch
Source: Gray Arial 10pt
top related