elasticsearch 20150107

PowerPoint Presentation

Lucene Query - Domain Specific Query -07/01/2015

Supership

QueryParser

Lucene

Lucene

Domain Specific Query

2

Supership

QueryParser

Lucene

Lucene

Domain Specific Query

3

GitHub @shingoOKAWATwitter @okawa_shingo

Supership 201511

(Elasticsearch 3 )/

()

4

Supership

QueryParser

Lucene

Lucene

Domain Specific Query

5

Supership7

3

2

2015113

Supership

QueryParser

Lucene

Lucene

Domain Specific Query

8

Query Parser Query Parser

(Proximity Search) (DisMax)

BNF DSQ(Domain Specific Query) (Concrete/Abstract Syntax Tree) API 9

Query Parser

UX()

10

Query Parser

PhraseQuery SpanQuery ()

Term

11PhraseQuery: cat dog bird, slop: 3SpanQuery: cat dog bird, slop: 3catbirdhorselizarddog

20catbirdhorselizarddog20

Query Parser (Hetero Grammar)

12

Query Parser ANTLR BNF

QueryParser.jj

13Query Query(String field) :{ Query q, firstQuery=null; int conj, mods;}{ mods=Modifiers() q=Clause(field) { addClause(clauses, CONJ_NONE, mods, q); if (mods == MOD_NONE) firstQuery=q; } : :grammar ExternalQuery;import CommonLexerRules;

query : (expression)+ ;

expression : CONJUNCTION_AND clause | CONJUNCTION_OR clause | clause ;:::

Query Parser API

Concrete Syntax Tree Abstract Syntax Tree

Term Synonym

14Concrete Syntax TreeORFFAbstract Syntax TreeBooleanClauseAT SAME POSITIONFFFinal Fantasy

TAG: BOOLEANTAG: TERMTAG: TERM

Supership

QueryParser

Lucene

Lucene

Domain Specific Query

15

Lucene Apache Lucene IR Library

Language Analysis (Filtering, Normalize, Tokenize)

Indexing and Storage (Lock-Free Indexing, Near-Real-Time Indexing)

Querying (50-over Query Implemantations, Parser, Scoring Model)

16Querying ElasticsearchQueryDSL

Lucene 17Documentfield1field2Text Analysis ChainsIndexWriterDocumentWriterIndexing ChainQueryParserQuery StringIndexReaderScorerQueryIndexing ChainSegmentPersistent LayerSegmentReaderTermsEnumPositionsEnum

Supership

QueryParser

Lucene

Lucene

Domain Specific Query

18

Lucene Apache Lucene Query API

(TermQuery, BooleanQuery)

(WildcardQuery, RegexpQuery)

(SpanQuery, PhraseQuery)

19

Lucene Apache Lucene Query API

(TermQuery, BooleanQuery)

(WildcardQuery, RegexpQuery)

(SpanQuery, PhraseQuery)

20Term

Lucene PhraseQuery

Term

Search Phase Analyzer 21cat dog bird, slop: 3catbirdhorselizarddog

2014Matched DocumentPhraseQuerycat

TermQueryTermQueryTermQuerydogbird

Lucene SpanNearQuery

SpanTerm

Search Phase Analyzer 22cat dog bird, slop: 3

22Matched DocumentSpanNearQuerycat

SpanTermQuerydogbirdSpanTermQuerySpanTermQuerycatbirdhorselizarddog20

Lucene SpanOrQuery

SpanQuery

BooleanClause Occurrence 23cat dog bird, slop: 3 OR cat dog bird, slop: 3

23Matched DocumentsSpanOrQuery

SpanNearQuerySpanNearQuerycatbirdhorselizarddog20catbirddog00

Supership

QueryParser

Lucene

Lucene

Domain Specific Query

24

Domain Specific Query ()

()

25{ "query": { "hetero_query": { "query": "this AND that", "handler_name": "external_mapper } }}{ "query": { "hetero_query": { "query": "this AND that", "handler_name": internal_mapper } }}

Domain Specific Query26grammar ExternalQuery;import CommonLexerRules;

query : (expression)+ ;

expression : CONJUNCTION_AND clause | CONJUNCTION_OR clause | clause ;

clause : MODIFIER_REQUIRE field | MODIFIER_NEGATE field | field ;

field : {_input.LT(2).getType() == COLON}? SINGLE_LITERAL COLON term | term ;

term : SINGLE_LITERAL # BareTerm | PHRASE_LITERAL # QuotedTerm ;

Domain Specific Query27grammar InternalQuery;import CommonLexerRules;

query : (expression)+ ;

expression : CONJUNCTION_AND clause | CONJUNCTION_DIS clause | CONJUNCTION_OR clause | LPAREN query RPAREN | clause ;

clause : MODIFIER_REQUIRE field (HAT SINGLE_LITERAL)? | MODIFIER_NEGATE field (HAT SINGLE_LITERAL)? | field (HAT SINGLE_LITERAL)? ;

field : {_input.LT(2).getType() == COLON}? SINGLE_LITERAL COLON term | term ;

term : SINGLE_LITERAL # BareTerm | PHRASE_LITERAL # QuotedTerm ;

Supership

QueryParser

Lucene

Lucene

Domain Specific Query

28

(Google)

Synonym Map AST API

Term AST API

Span

Query Parser 29