형태소 분석기를 적용한 elasticsearch 운영

elasticsearch

Docker Seoul Meetup #4, 2015 1

N42@seapy

Docker Seoul Meetup #4, 2015 2

elasticsearch AWS Docker

Docker Seoul Meetup #4, 2015 3

...

Docker Seoul Meetup #4, 2015 4

elasticsearch,

Docker Seoul Meetup #4, 2015 5

Docker Seoul Meetup #4, 2015 6

! elasticsearch

Docker Seoul Meetup #4, 2015 7

Docker Seoul Meetup #4, 2015 8

! elasticsearch+

!

Docker Seoul Meetup #4, 2015 9

!1

1 http://eunjeon.blogspot.kr/

Docker Seoul Meetup #4, 2015 10

2

mecab-ko :

mecab-ko-dic :

mecab-java :

elasticsearch mecab-ko

2 (elasticsearch) (eunjeon) - nacyot

Docker Seoul Meetup #4, 2015 11

http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/

$ tar zxfv mecab-ko-XX.tar.gz$ cd mecab-ko-XX$ ./configure && make && make check$ sudo make install

$ tar zxfv mecab-ko-dic-XX.tar.gz$ cd mecab-ko-dic-XX$ ./configure && make$ sudo make install

#...

2 2 (elasticsearch) (eunjeon) - nacyot

Docker Seoul Meetup #4, 2015 12

http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/

automake , .

libmecab.so.2 , .

!Docker Seoul Meetup #4, 2015 13

elasticsearch analyzer

!

Docker Seoul Meetup #4, 2015 14

!

Docker Seoul Meetup #4, 2015 15

''

,,,,NNP,*,F,,*,*,*,*

Docker Seoul Meetup #4, 2015 16

mecab-ko-dic

Docker Seoul Meetup #4, 2015 17

or Docker Seoul Meetup #4, 2015 18

Docker Seoul Meetup #4, 2015 19

,

Docker Seoul Meetup #4, 2015 20

Docker

Docker Seoul Meetup #4, 2015 21

Docker $ docker run -d --name elasticsearch \ -v /home/data-xxx:/data \ -p 9200:9200 n42corp/elasticsearch-n42

+ elasticsearch 3

/ 4

4 https://github.com/n42corp/search-ko-dic

3 https://github.com/nacyot/elasticsearch

Docker Seoul Meetup #4, 2015 22

head, kopf

Docker Seoul Meetup #4, 2015 23

ShieldSecurity for Elasticsearch

Docker Seoul Meetup #4, 2015 24

Nginx

server { listen *:80;

location / { auth_basic "ElasticSearch"; auth_basic_user_file /etc/nginx/search.htpasswd;

try_files @elasticsearch @elasticsearch; }

location @elasticsearch { proxy_pass http://xxxx.com:9200; }}

Docker Seoul Meetup #4, 2015 25

Dockerizing

$ docker run -d -p 80:80 \ --link elasticsearch \ -e SEARCH_USER='xxx' \ -e SEARCH_PASSWORD_ENCRYPTED='encrypted password' \ n42corp/elasticsearch-proxy-nginx$ curl -XGET http://xxx:passwd@localhost

Docker Seoul Meetup #4, 2015 26

Docker Composenginx, elasticsearch

Docker Seoul Meetup #4, 2015 27

docker-compose.ymlelasticsearch: image: n42corp/elasticsearch volumes: - /home/data:/datanginx: image: n42corp/elasticsearch-proxy-nginx links: - elasticsearch ports: - "80:80" environment: - SEARCH_USER=username - SEARCH_PASSWORD_ENCRYPTED=$apr1$o2nD6/0t$U6DaCfEqLaIZptGKYw84Y.

Docker Seoul Meetup #4, 2015 28

$ docker-compose up -d$ curl -XGET http://username:password@192.168.99.100/

{ "status" : 200, "name" : "Fusion", "cluster_name" : "elasticsearch", "version" : { "number" : "1.7.0", "build_hash" : "929b9739cae115e73c346cb5f9a6f24ba735a743", "build_timestamp" : "2015-07-16T14:31:07Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search"}$ docker-compose stop

Docker Seoul Meetup #4, 2015 29

Production

https://github.com/docker/compose

Docker Seoul Meetup #4, 2015 30

Docker Seoul Meetup #4, 2015 31

Docker Seoul Meetup #4, 2015 32

ONBUILDn42corp/elastcisearch Dockerfile

#... # ONBUILD COPY servicecustom.csv /opt/mecab-ko-dic-2.0.1-20150707/user-dic/servicecustom.csvONBUILD RUN cd /opt/mecab-ko-dic-2.0.1-20150707 &&\ tools/add-userdic.sh &&\ make install

# ONBUILD COPY synonym.txt /elasticsearch/config/synonym.txt#...

Docker Seoul Meetup #4, 2015 33

Dockerfile (servicecustom.csv), (synonym.txt)4 Dockerfile

FROM n42corp/elasticsearch

$ docker build -t n42corp/elasticsearch-n42 .

4 https://github.com/n42corp/search-ko-dic

Docker Seoul Meetup #4, 2015 34

elasticsearch

Docker Seoul Meetup #4, 2015 35

Docker Seoul Meetup #4, 2015 36

es

Docker Seoul Meetup #4, 2015 37

Docker ? Docker

Docker Seoul Meetup #4, 2015 38

IP

$ docker run -d -p 9200:9200 -p 9300:9300 \ -v /home/data-xxx:/data \ n42corp/elasticsearch \ --cluster.name=pangyo_market \ --node.name=$(hostname) \ --network.publish_host=$(hostname -i) \ --discovery.zen.ping.multicast.enabled=false \ --discovery.zen.ping.unicast.hosts=x.x.x.x:9300,y.y.y.y:9300

Docker Seoul Meetup #4, 2015 39

AWS 5

$ docker run -d --name elasticsearch \ -v /home/data-xxx:/data \ -e AWS_ACCESS_KEY_ID=xxxx \ -e AWS_SECRET_KEY=yyyy \ -p 9200:9200 \ n42corp/elasticsearch \ --cluster.name=pangyo_market \ --node.name=$(hostname) \ --discovery.type=ec2

5 https://github.com/elastic/elasticsearch-cloud-aws

Docker Seoul Meetup #4, 2015 40

Docker Seoul Meetup #4, 2015 41

,

haproxy, nginx, ELB

Docker Seoul Meetup #4, 2015 42

AWS ECS

ECS

ELB

ECS OS ECS

+ kubenertes, fleet

Docker Seoul Meetup #4, 2015 43

docker-machine & docker-compose

(docker-machine)

yaml (docker-compose)

docker-compose production

Docker Seoul Meetup #4, 2015 44

Docker Seoul Meetup #4, 2015 45

API

Docker Seoul Meetup #4, 2015 46

sinatra dockerizing-

$ docker run -d -p 4567:4567 \ n42corp/korean-morpheme-sinatra

# posids 150(),151() $ curl -XGET 'http://192.168.59.103:4567/morpheme' \ -d 'text= &posids=150,151'

- https://github.com/n42corp/dockerfiles/tree/master/korean-morpheme-sinatra

Docker Seoul Meetup #4, 2015 47

{ "morps": [ { "surface": "", "posid": 150, "desc": " ", "feature": "NNG,*,T,,*,*,*,*" }, { "surface": "", "posid": 151, "desc": " ", "feature": "NNP,*,F,,*,*,*,*" }, { "surface": "", "posid": 150, "desc": " ", "feature": "NNG,*,F,,*,*,*,*" } ]}

Docker Seoul Meetup #4, 2015 48

RORLab

curl -XGET 'http://rorla2.rorlab.org:4567/morpheme' \ -d 'text= &posids=150,151'

Docker Seoul Meetup #4, 2015 49