an overview of similarity query processing

An Overview of Similarity Query Processing

2014. 2. 26.김종익

전북대학교 컴퓨터공학부

2

Table of Contents

01. Applications of similarity query processing 02. Problem Formulation03. string Decomposition04. Similarity Function05. A naïve approach06. Overlap Similarity07. Similarity Query Processing with Inverted lists08. Similarity Function Revisited09. Filter and Verification Framework10. Prefix Filtering based Approach11. Exploiting Document Frequency Ordering

3

Some examples and figures in this presentationare taken from the following materials

Marios Hadjieleftheriou and Chen Li, Efficient Approximate Search on String Collections (tutorial), ICDE 2009 and VLDB 2009

Chuan Xiao, Wei Wang, Xuemin Lin, Jeffrey Xu Yu, Efficient Similarity Joins for Near Duplicate Detection, WWW 2008 (slide)

Jongik Kim and Hongrae Lee, Efficient Exact Similarity Searches using Multiple Token Orderings, ICDE 2012 (slide)

Applications of similarity query processing (1/8)

4

Actual queries gatheredby Google

Web Search

5

Should be “Niels Bohr”


Data Integration and data cleaning

Rinformix … …microsoft … …… … …… … …

Sinfromix …

… …

mcrosoft …

… …

6


Duplicate (Web) Documents Detection

7


Identify Spams

SPAM TEMPLATE

Sir/Madam,We happily announce to you the draw of the EURO MILLIONS SPANISH LOTTERY INTERNATIONALWINNINGS PROGRAM PROMOTIONS held on the 27TH MARCH 2008 in SPAIN. Your company or yourpersonal e-mail address attached to ticket number 653-908-321-675 with serial main number <NUMBER> drew lucky star winning numbers <NUMBER> which consequently won in the 2ND category, you have therefore been approved for a lump sum pay out of 960.000.00 Euros. (NINE HUNDRED AND SIXTY THOUSAND EUROS).CONGRATULATIONS!!!

Sincerely yours,<NAME><AFFILIATION>

8


Detect Plagiarism

Q. What are the advantages of RAID5 over RAID4?A. 1. Several write requests could be processed in parallel, since the bottleneck of a unique check disk has been eliminated. 2. Read requests have a higher level of parallelism. Since the data is distributed over all disks, read requests involve all disks, whereas in systems with a dedicated check disk the check disk never participates in read.

Q. What are the advantages of RAID5 over RAID4? A. 1. Several write requests could be processed in parallel, since the bottleneck of a single check disk has been eliminated. 2. Read requests have a higher level of parallelism on RAID5. Since the data is distributed over all disks, read requests involve all disks, whereas in systems with a check disk the check disk never participates in read.

Recommendation of friends in an SNS service

9


Friends vector: 1 0 0 1 1 0 0 1 Friends vector: 1 0 0 1 1 1 0 1

Friends of a person can be representation of a binary vector

Read (a fragment of genome sequence) Alignment

10


GCTGATGTGCCGCCTCACTCCGGTGG …

CACTCCTGTGGCTCACTCCTGTGG

GCTGATGTGCCACCTCAGATGTGCCACCTCACTC

GTGCCGCCTCACTCCTGCTCCTGTGG

Reference sequence

Short reads

11


Supported by Oracle Text CREATE TABLE engdict(word VARCHAR(20), len INT); Create preferences for text indexing:

begin ctx_ddl.create_preference('STEM_FUZZY_PREF', 'BASIC_WORDLIST'); ctx_ddl.set_attribute('STEM_FUZZY_PREF','FUZZY_MATCH','ENGLISH'); ctx_ddl.set_attribute('STEM_FUZZY_PREF','FUZZY_SCORE','0'); ctx_ddl.set_attribute('STEM_FUZZY_PREF','FUZZY_NUMRESULTS','5000'); ctx_ddl.set_attribute('STEM_FUZZY_PREF','SUBSTRING_INDEX','TRUE'); ctx_ddl.set_attribute('STEM_FUZZY_PREF','STEMMER','ENGLISH'); end; /

CREATE INDEX fuzzy_stem_subst_idx ON engdict ( word ) INDEXTYPE IS ctxsys.context PARAMETERS ('Wordlist STEM_FUZZY_PREF');

Usage:SELECT * FROM engdict WHERE CONTAINS(word, 'fuzzy(universisty, 70, 6, weight)', 1) > 0;

Limitation: cannot handle errors in the first letters:Katherine versus Catherine

Query Relaxation

12

Problem Formulation (1/2)

Find strings similar to a given string

Similar to: a domain-specific function returns a similarity value between two strings

Common similarity functions: Jaccard coefficient Cosine similarity Dice similarity Edit distance

13

Problem Formulation (2/2)

Functions require set data

14

String DecompositionWord tokens for long string (e.g. web page)

x = “yes as soon as possible”y = “as soon as possible please”

x = {A, B, C, D, E}y = {B, C, D, E, F}

word yes as soon as1 possbile please

token A B C D E F

q-gram tokens for short string (e.g. keyword query)x = “universal”

G(x, 2) = {un, ni, iv, ve, er, rs, sa, al}

u n i v e r s a l

15

Similarity Function

Jaccard Similarity

Cosine similarity

Dice similarity

4( , ) 0.676

x yJ x y

x y

x = {A, B, C, D, E}y = {B, C, D, E, F}

| | 4( , ) 0.85| || |

x yC x yx y

2 | | 8( , ) 0.8| | | | 10

x yD x yx y

Edit DistanceED(x, y) = minimum number of edit operations to change x to y (insertion, deletion, substitution)

x: Tom Hanks y: Ton Hank ED(x, y) = 2

16

A naïve approach

Given a collection of strings C, a query string x, and a threshold t of a similarity function sim,

1. decompose each string in C and the query string into tokens.2. output those string y ∈ C such that sim(x, y) ≥ t.

Since C contains a lot of strings, this approach is obviously inefficient.

17

Overlap Similarity (1/2)

yxt

tyxO

1

),(tyxyx

yxJ

),(

Given a similarity threshold t,

| | ( , )| | | | | | | | | | ( , )

x y O x y tx y x y x y O x y

( , ) | || |O x y t x y | |( , )| || |x yC x y tx y

2 | |( , )| | | |

x yD x y tx y

(| | | |)( , )

2t x yO x y

( , ) 4O x y x y Overlap Similarity

18

Overlap Similarity (2/2)

( ( , ), ( , )) max(| ( , ) |,| ( , ) |)O G x q G y q G x q G y q d q ( , )ED x y d

Given an edit distance d,

u n i v e r s a l

d edit operations could affect d x q gramsor, d edit operations on x can mutate d x q grams of x

x = “universal” and G(x, 2) = {un, ni, iv, ve, er, rs, sa, al}2 edit operations on x mutate 2 x 2 q-gramsHence, y should contains at least |G(x, 2)| - 2 x 2 = 4 q-grams in G(x, 2)

19

Similarity Query Processing with Inverted lists

ID String Record (token set)1 area { , re, ea}2 artisan { , rt, ti, is, sa, an}3 artist { , rt, ti, is, st}4 tisk {ti, is, sk}… … …

arskeais

sart

stti

1 2 3412

22 3

32 3 4

re 1Make Inverted Lists

an 2

3

Query: “artist” Overlap threshold: 4

Merge to count occurrences

1

2

3

4

2

4

5

2

Answers of the query 2: “artisan” 3: “artist”

{ , , , , }ar rt ti is st

4ararar

20

1234

1

3

37

3 2

Count threshold t≥ 3

minHeap

21717

1: count 2 < t (X)2: count 3 = t (O)

…11 2

3 3

22

Merge Algorithm – HeapMerge

Similarity Function Revisited

( , ) | |O x y t x tyxyx

yxJ

),(

Given a query x with a similarity threshold t, FOR ALL y,

| | | | (| | | |)1

| | | |

ty x y x yt

y t x

2( , ) | |O x y t x | |( , )| || |x yC x y tx y

2 | |( , )| | | |

x yD x y tx y

| |( , )

2t xO x y

t

To determine the overlap threshold, we need to know the size of y, whichvaries according to each string in a collection.

( ( , ), ( , )) | ( , ) |O G x q G y q G x q t q ( , )ED x y t

22

Filter and Verification Framework

Find those strings that shares at least α tokens with the query string, where α is an overlap lower bound.

FILTER

Verify each string found in filtering stage by directly applying a similarity function

VERIFICATION

Quickly generate initial candidates using a minimum constraint

Refine candidates using α

FILTER REFINEMENT

23

Prefix Filtering based ApproachQuery x = “artist” {ar, rt, ti, is, st} and overlap threshold α = 4

arisrtstti

1 2 322 332 3 4

3

Inverted lists for the query

strt

332

aris

ti1 2 32

2 3 4

3

Sort the listsby their sizes

Prefix Lists: the first |G(x, 2)| – α + 1 lists

Suffix Lists: remaining α – 1 lists

Filtering Phase (the prefix filtering) Merge the prefix lists to generate candidates

Refinement Phase Search the suffix lists for each candidate A candidate searches each suffix list to identify if it is contained in the list Binary search is used because suffix lists are usually very long

23

12

candidates 2 3 43 4 5

4

4Sort the tokens by theirdocument frequencies

Document frequencyordering

24

Exploiting Document Frequency Ordering (1/2) General Goal: minimize the number of candidates initially generated

by making use of the document frequency ordering

rtstti

2 332 3 4

aris

1 2 32 3 4

strt

332

aris

ti1 2 32

2 3 4

3 4


Query x = “artist” {ar, rt, ti, is, st} and overlap threshold α = 4




Sort the tokens by theirdocument frequencies

234

candidates

123

candidates

We can reduce1. time for merging short lists2. number of candidates

time for verification candidates

25

Query x = {w1, w2} and overlap threshold α = 2

w2 is the prefix list# of candidates is 5



Total number of candidates is 0

Partition

Observation By partitioning a data set, we can artificially modify document frequencies of tokens in

each partition. We evaluate a query in each partition and take the union of the results. We can reduce the number of candidates by utilizing different token orderings among

partitions. Because partitions have different token orderings, we need to sort tokens in a query record

in each partition.

Exploiting Document Frequency Ordering (2/2)