형태소 분석기를 적용한 elasticsearch 운영

Download 형태소 분석기를 적용한 elasticsearch 운영

Post on 15-Apr-2017

5.804 views

Category:

Software

6 download

Embed Size (px)

TRANSCRIPT

  • elasticsearch

    Docker Seoul Meetup #4, 2015 1

  • N42@seapy

    Docker Seoul Meetup #4, 2015 2

  • elasticsearch AWS Docker

    Docker Seoul Meetup #4, 2015 3

  • ...

    Docker Seoul Meetup #4, 2015 4

  • elasticsearch,

    Docker Seoul Meetup #4, 2015 5

  • Docker Seoul Meetup #4, 2015 6

  • ! elasticsearch

    Docker Seoul Meetup #4, 2015 7

  • Docker Seoul Meetup #4, 2015 8

  • ! elasticsearch+

    !

    Docker Seoul Meetup #4, 2015 9

  • !1

    1 http://eunjeon.blogspot.kr/

    Docker Seoul Meetup #4, 2015 10

  • 2

    mecab-ko :

    mecab-ko-dic :

    mecab-java :

    elasticsearch mecab-ko

    2 (elasticsearch) (eunjeon) - nacyot

    Docker Seoul Meetup #4, 2015 11

    http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/

  • $ tar zxfv mecab-ko-XX.tar.gz$ cd mecab-ko-XX$ ./configure && make && make check$ sudo make install

    $ tar zxfv mecab-ko-dic-XX.tar.gz$ cd mecab-ko-dic-XX$ ./configure && make$ sudo make install

    #...

    2 2 (elasticsearch) (eunjeon) - nacyot

    Docker Seoul Meetup #4, 2015 12

    http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/http://blog.nacyot.com/articles/2015-06-13-eunjeon-with-elasticsearch/

  • automake , .

    libmecab.so.2 , .

    !Docker Seoul Meetup #4, 2015 13

  • elasticsearch analyzer

    !

    Docker Seoul Meetup #4, 2015 14

  • !

    Docker Seoul Meetup #4, 2015 15

  • ''

    ,,,,NNP,*,F,,*,*,*,*

    Docker Seoul Meetup #4, 2015 16

  • mecab-ko-dic

    Docker Seoul Meetup #4, 2015 17

  • or Docker Seoul Meetup #4, 2015 18

  • Docker Seoul Meetup #4, 2015 19

  • ,

    Docker Seoul Meetup #4, 2015 20

  • Docker

    Docker Seoul Meetup #4, 2015 21

  • Docker $ docker run -d --name elasticsearch \ -v /home/data-xxx:/data \ -p 9200:9200 n42corp/elasticsearch-n42

    + elasticsearch 3

    / 4

    4 https://github.com/n42corp/search-ko-dic

    3 https://github.com/nacyot/elasticsearch

    Docker Seoul Meetup #4, 2015 22

  • head, kopf

    Docker Seoul Meetup #4, 2015 23

  • ShieldSecurity for Elasticsearch

    Docker Seoul Meetup #4, 2015 24

  • Nginx

    server { listen *:80;

    location / { auth_basic "ElasticSearch"; auth_basic_user_file /etc/nginx/search.htpasswd;

    try_files @elasticsearch @elasticsearch; }

    location @elasticsearch { proxy_pass http://xxxx.com:9200; }}

    Docker Seoul Meetup #4, 2015 25

  • Dockerizing

    $ docker run -d -p 80:80 \ --link elasticsearch \ -e SEARCH_USER='xxx' \ -e SEARCH_PASSWORD_ENCRYPTED='encrypted password' \ n42corp/elasticsearch-proxy-nginx$ curl -XGET http://xxx:passwd@localhost

    Docker Seoul Meetup #4, 2015 26

  • Docker Composenginx, elasticsearch

    Docker Seoul Meetup #4, 2015 27

  • docker-compose.ymlelasticsearch: image: n42corp/elasticsearch volumes: - /home/data:/datanginx: image: n42corp/elasticsearch-proxy-nginx links: - elasticsearch ports: - "80:80" environment: - SEARCH_USER=username - SEARCH_PASSWORD_ENCRYPTED=$apr1$o2nD6/0t$U6DaCfEqLaIZptGKYw84Y.

    Docker Seoul Meetup #4, 2015 28

  • $ docker-compose up -d$ curl -XGET http://username:password@192.168.99.100/

    { "status" : 200, "name" : "Fusion", "cluster_name" : "elasticsearch", "version" : { "number" : "1.7.0", "build_hash" : "929b9739cae115e73c346cb5f9a6f24ba735a743", "build_timestamp" : "2015-07-16T14:31:07Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search"}$ docker-compose stop

    Docker Seoul Meetup #4, 2015 29

  • Production

    https://github.com/docker/compose

    Docker Seoul Meetup #4, 2015 30

  • Docker Seoul Meetup #4, 2015 31

  • Docker Seoul Meetup #4, 2015 32

  • ONBUILDn42corp/elastcisearch Dockerfile

    #... # ONBUILD COPY servicecustom.csv /opt/mecab-ko-dic-2.0.1-20150707/user-dic/servicecustom.csvONBUILD RUN cd /opt/mecab-ko-dic-2.0.1-20150707 &&\ tools/add-userdic.sh &&\ make install

    # ONBUILD COPY synonym.txt /elasticsearch/config/synonym.txt#...

    Docker Seoul Meetup #4, 2015 33

  • Dockerfile (servicecustom.csv), (synonym.txt)4 Dockerfile

    FROM n42corp/elasticsearch

    $ docker build -t n42corp/elasticsearch-n42 .

    4 https://github.com/n42corp/search-ko-dic

    Docker Seoul Meetup #4, 2015 34

  • elasticsearch

    Docker Seoul Meetup #4, 2015 35

  • Docker Seoul Meetup #4, 2015 36

  • es

    Docker Seoul Meetup #4, 2015 37

  • Docker ? Docker

    Docker Seoul Meetup #4, 2015 38

  • IP

    $ docker run -d -p 9200:9200 -p 9300:9300 \ -v /home/data-xxx:/data \ n42corp/elasticsearch \ --cluster.name=pangyo_market \ --node.name=$(hostname) \ --network.publish_host=$(hostname -i) \ --discovery.zen.ping.multicast.enabled=false \ --discovery.zen.ping.unicast.hosts=x.x.x.x:9300,y.y.y.y:9300

    Docker Seoul Meetup #4, 2015 39

  • AWS 5

    $ docker run -d --name elasticsearch \ -v /home/data-xxx:/data \ -e AWS_ACCESS_KEY_ID=xxxx \ -e AWS_SECRET_KEY=yyyy \ -p 9200:9200 \ n42corp/elasticsearch \ --cluster.name=pangyo_market \ --node.name=$(hostname) \ --discovery.type=ec2

    5 https://github.com/elastic/elasticsearch-cloud-aws

    Docker Seoul Meetup #4, 2015 40

  • Docker Seoul Meetup #4, 2015 41

  • ,

    haproxy, nginx, ELB

    Docker Seoul Meetup #4, 2015 42

  • AWS ECS

    ECS

    ELB

    ECS OS ECS

    + kubenertes, fleet

    Docker Seoul Meetup #4, 2015 43

  • docker-machine & docker-compose

    (docker-machine)

    yaml (docker-compose)

    docker-compose production

    Docker Seoul Meetup #4, 2015 44

  • Docker Seoul Meetup #4, 2015 45

  • API

    Docker Seoul Meetup #4, 2015 46

  • sinatra dockerizing-

    $ docker run -d -p 4567:4567 \ n42corp/korean-morpheme-sinatra

    # posids 150(),151() $ curl -XGET 'http://192.168.59.103:4567/morpheme' \ -d 'text= &posids=150,151'

    - https://github.com/n42corp/dockerfiles/tree/master/korean-morpheme-sinatra

    Docker Seoul Meetup #4, 2015 47

  • { "morps": [ { "surface": "", "posid": 150, "desc": " ", "feature": "NNG,*,T,,*,*,*,*" }, { "surface": "", "posid": 151, "desc": " ", "feature": "NNP,*,F,,*,*,*,*" }, { "surface": "", "posid": 150, "desc": " ", "feature": "NNG,*,F,,*,*,*,*" } ]}

    Docker Seoul Meetup #4, 2015 48

  • RORLab

    curl -XGET 'http://rorla2.rorlab.org:4567/morpheme' \ -d 'text= &posids=150,151'

    Docker Seoul Meetup #4, 2015 49