elasticsearch x autoscaling(aws)

Download Elasticsearch x Autoscaling(AWS)

Post on 10-Aug-2015

1.273 views

Category:

Engineering

4 download

Embed Size (px)

TRANSCRIPT

  1. 1. Amazon Web Searvice elasticsearch
  2. 2. DMPAudienceSearch
  3. 3. IMelasticsearch elasticsearch AutoScaling
  4. 4. elasticsearch ID User Agent IP ID
  5. 5. AudienceSearch 1000 30
  6. 6. AWS Cluster: 1, Type: 1, Node: 86 master: 3 (master:true, data: false) data: 80 (master:false, data:true) searcher: 2 (master:false, data: false) indexer: 1 (master:false, data: false) version: 1.3.8 1.5.2 (5/26)
  7. 7. masterdatasearcher x2 indexer indexerAutoScaling x3x80 ELB
  8. 8.
  9. 9. Cluster Size Shard = Node 10 200 100 200Aggregation 80
  10. 10. Shard Node = 80 c3.large (1) shard: 40, replica: 1 shard/node = 0.5 (2) shard: 80, reprica: 1 shard/node = 1 (3) shard: 160, replica:1 shard/node = 2 shard/node shard/nodeshard/cpu
  11. 11. Instance Size SSD Ephemeral Diskc3 xlargeCP largexlarge30GB80GB JVMdoc_value
  12. 12. AutoScalling
  13. 13. Why AutoScaling ? AutoScaling terminate
  14. 14. recovery LifeCycle recovery 0 indices: recovery: max_bytes_per_sec: 500mb concurrent_streams: 4
  15. 15. recovery LifeCycle : 40GB 10c3.xlarge
  16. 16. elasticsearch-cloud-aws discoverymaster node SG discovery: type: ec2 ec2: tag: es_cluster: production
  17. 17. master node aws:autoscaling:groupName
  18. 18. ELB ELB master, data 92009300
  19. 19.
  20. 20. Multi-AZ3 shardreplicaAZ cluster: routing: allocation: awareness: attributes: aws_availability_zone
  21. 21. aws_availability_zoneAZ elasticsearch-cloud-aws { "attributes" : { "aws_availability_zone" : "ap-northeast-1c", "master" : "false" } }
  22. 22. hostname hostname(ip-10-0-0-1)ES AMI node: name: ${HOSTNAME} export HOSTNAME=$(hostname -s) /etc/syscong/elasticsearch /etc/elasticsearch/elasticsearch.yml
  23. 23. ELB Mackerel Papertrail /Slackwebhook
  24. 24. Papertrail master data searcher x2 x3 x80 SSHtail syslog
  25. 25. AWSES AutoScalling Cluster Size 1indexelasticsearch
  26. 26.
  27. 27.