elastic stackにハマった話

Been Stuck In Elastic StackKazuhiro Kosaka

Engineer, MDH, CyberAgent, Inc.

Feb 10, 2017

About Me• 小坂和弘 - Kazuhiro Kosaka @hyperdash• 2009- CyberAgent, Inc.• Designer/Front-end Engineer/Back-end Engineer• Motion Graphics, VJ

About Me• I love hiking

Summary

What To Talk?• Amazon Elasticsearch Service• v1.x v2.x

Summary

What NOT To Talk?• Private Elasticsearch Cluster / Elastic Cloud• v5.x

Summary

Elastic Stack?• Had been called ELK Stack

• Elasticsearch - Distributed Search/Analytics Engine• Logstash - Data Processing Pipeline• Kibana - Data Visualizer• Beats - Data Shipping Agents

Our Case広告運用・配信設定管理画面ユーザ操作ログ

Our Case• Elasticsearch v1.5 on Amazon Elasticsearch Service• Logstash v2.3.1• Filebeat v1.2.1• m3.large / 3 instances• 400,000 docs / day

• 95% of logs are by 日予算自動アロケーション Batch• Will be separated into another index

• 主にユーザの操作履歴調査

Our Case

Stuck #1 / SettingsDataLoss を防ぐには？

• Filebeat uses a back-pressure sensitive protocol• Logstash < v5.1 has NO Persistent Queue


Source: https://www.elastic.co/guide/en/logstash/5.0/deploying-and-scaling.html

https://www.elastic.co/guide/en/logstash/5.0/deploying-and-scaling.html


• Use Message Queue e.g. Kafka• Delete Filebeat registrar file and Re-ship data

• >= v5.1 Persistent Queue (beta) is enabled

Stuck #2 / JVMMemoryPressureES に Write できなくなったんですけど…

• m3.medium / 1 instance• JVMMemoryPressure ≒ 92-94 %• FreeStorageSpace looks okay


Source: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html

• NOT a t2 instance• ???

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html


m3.medium / 1 instance

m3.large / 3 instances

Worked!


• indices.fielddata.cache.size: unbounded (default)This allows ES to allocate unlimited memory for fielddata

• indices.fielddata.cache.size: 50 (%)


Source: https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html

https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html


• ES v1.x• Doc Values: Default OFF• Set “doc_values”: true in index template - mappings


• ES >= v2.x• Doc Values: Default ON

Stuck #3 / RestrictionsAmazon Elasticsearch Service の制限

• Dynamic Scripting is not supported• Custom Plugins are not supported• Access Control

• Amazon ES is out of VPC• IP-Based, IAM-Based Access Control• logstash-output-amazon_es plugin is not stable??

Next Step

• Migrate to Elastic Stack v5.x• Elastic Cloud or Private Cluster?

• 管理画面上での操作ログ閲覧

Conclusion

• Amazon Elasticsearch Service

• Elastic Cloud• Easy & Handy (Fully-Managed)• The latest version is available anytime• Less Restrictions, or Elastic Cloud Enterprise (alpha)• Supported by Elastic

• Easy & Handy (Fully-Managed)• Restrictions• Supported by AWS

END

THANK YOU

elastic stackにハマった話

Software