By Floris Dekker – September 2nd, 2013Click to edit Master title styleBy Tobias Rein & Alex Eftimie – October 12th, 2016Elastic @ GetYourGuide

5 students wanted to build apeer-to-peer platform for tour guides.

But we also learned one thing:

There is a greater problem to be solved.

So the team started from scratch . . .

. . . and we relaunched

is me e too o

e ee p e

e

1

We built a unique culture.

@GetYourGuide

In 2012 …

… we turned MySQL into elasticsearch

0.19.80.90.01.0.02.3.1

Upgrading was usually painless, thanks to Elastica (A PHP client for elasticsearch.)

MySQL is still our primary storage.

We are using asynchronous messages to update the elasticsearch index

Our Setup

Our architecture

1. Location lookup[elasticsearch]

2. Tour search[elasticsearch]

3. Availability filter[MySQL]

Multi-step

Our Cluster

56 indexesfor customer search

26 indexesfor intranet

4 indexes in 14 languages up to 2.5 million docs per index

3 nodes 8.1 GB

2 replicas, no sharding 10 million documents

Our migration strategy

• Unique index namesusing timestamp

• An alias <name>_live points to the current index

• For migration we use temporary alias names <name>_temp

@GetYourGuide

ELK

What do we do with logs @ GYG ?

• Debugging • Business Intelligence

Motivation for a new infrastructure

• SSH to host no longer an option

• Local storage in the cloud is limited

• No one uses outdated tools

• Fault tolerant trustworthy logs are a must!

• Redundancy, Auto failover, Monitoring

Logging infrastructure - FLKLEK

Two m4.2xlarge AWS instances

(One tie breaker instance in Digitalocean)

The Numbers

120 GB 20 Mioelasticsearch store size log lines /day

42 5.5Knodes avg. logs /second

Lessons learned

• Large documents affect ES performance

• Dynamic mappings are a pain

• Beats don’t play well with file rotation

• Kibana is a powerful tool

29

Questions?