Succinct Data Structure for Analyzing Document Collection
Preferred Infrastructure Inc.
Daisuke Okanohara
2011/12/1 ALSIP2011@Takamatsu
Agenda
Succinct Data Structures (20 min.) Rank/Select, Wavelet Tree
Full-text Index (30 min.) Suffix Array/Tree, BWT
Compressed Full-Text Index (20 min.) FM-index
Analyzing Document Collection (20 min.) Document Statistics (for mining) Top-k query
Succinct data Structures
Full-textIndex
Compressed Full-text Index
Analyzing Document Collection
Background
Very large document collections appear everywhere Twitter : 200 billion tweets per year. (6K tweets per sec.) Modern DNA Sequencing
E.g. Illumina HiSeq 2000 produces 40G bases per day
Analyzing these large document collections require new data
structures and algorithms
Notation
T[0, n] : string T[0]T[1]…T[n] T[i, j] : subarray from T[i] to T[j] inclusively T[i, j) : subarray from T[i] to T[j-1]
lg m : log2 m (# of bits to represent m cases)
Σ : the alphabet set of string σ := |Σ| : the number of alphabets n : length of input string m : length of query string
PreliminariesEmpirical Entropy (1/2) Empirical entropy indicates the compressibility of the string Zero-order empirical entropy of T H0(T) := ∑c n(c)/n log (n/p(c))
n(c) : the number of occurrences of c in T n : the length of T The lower-bound for the average code length of compressor not
using context information
T = abccbaabcab,
n(a)=4,
n(b)=4, n(c)=3, H0(T) = 1.57 bit
Empirical Entropy (2/2)
k-th order empirical entropy of T Hk(T) = ∑w∈Σ
k-1 nw/n H0(Tw) : Tw : the characters followed by a substring w nw : the number of occurrences of a string w in T.
The lower-bound for the average code length of compressor using
the following substring of length k-1 as a context information
T = abccbaabcabTa=bac Tb=acaa Tc=bcb, H2(T) = 0.33 bit
Hierarchy of entropy: log σ ≧ H0(T) ≧ H1(T) ≧ … Hk(T) Typical English Text, lg σ=8 H0=4.5 , H5=1.7
Succinct Data Structure
Rank/Select Dictionary
For a bit vector B[0, n), B[i] ∈{0, 1}, consider following operations Lookupc(B, i) : B[i] Rankc(B, i) : The number of c’s in B[0…i) Selectc(B, i) : The (i+1)-th occurrence of c in B
Example:i 0123456789B 0111011001
Rank1(B,6)=4 Select0(B,2)=2
The number of 1’s in B[0, 6) The (2+1)-th occurrence of 0 in B
Rank/Select Dictionary Usage (1/2)Subset For U = {0, 1, 2, …, n-1}, a subset of U, V⊆U can be represented
by a bit vector B[0, n-1], s.t. B[i] = 1 if i∈V, and B[i]=0
otherwise. Select(B, i) is the (i+1)-th smallest element in V, Rank(B, i) = |{e|e∈V, e<i}|
V = {0, 4, 7, 10, 15}
i 0123456789012345 B = 1000100100100001
Rank/Select Dictionary Usage (2/2)Partial Sums For a non-negative integer array P[0, m), we concatenate each
unary codes of P[i] to build a bit array B[0, n) An unary code of a non-negative integer p≧0 is 0p1
E.g. unary codes of 0, 3, 10 are 1, 0001, 00000000001 From the definition, n=m+∑p[i] and the number of ones is m
P[i] = Select1(B, i) - Select1(B, i-1). Cum[k] = ∑i=0…kP[i]. = Select1(B, k) - k (# of 0’s before (k+1)-th
1) Find r s.t. Cum[r] ≦ x < Cum[r+1] by Rank1(B, Select0(B, x))
P = {3, 5, 2, 0, 4, 7}
B = 000100000100110000100000001
Cum[4] = Select(B, 4) – 4 = 14
Data structures for Rank/Select
Bit vector B[0, n) with m ones can be represented in lg (n, m) bits lg (n, m) ≒ m lg2e + m lg (n/m) when m << n
The implementations of Rank/Select Dictionary are divided into
the case when the bit vector is dense, sparse, and very sparse.
Case 1. Dense Case 2. Sparse (RRR) Case 3. Very Sparse (SDArray)
01101011010100110101100101
00001001000010001000100010
00000000000010000000001000
Rank/Select DictionaryCase 1. Dense Divide B into large blocks of length l=(log n)2
Divide each large block into small blocks of length s=(log n)/2 For each large block, store L[i]=Rank1(B, il) For each small block, store S[i]=Rank1(B, is) - Rank1(B, is / l) Build a table P(X) that stores the number of ones in X of length s
Rank1(B, i) = L[i/l] + S[i/s] + P(B[i/s*s, i)) three table lookup. O(1) time
Total size of L, S, P is o(n)
Select can also be supported in O(1) time
using n+o(n) bits
L[1]
+S[4]
+P[X]
In practice, P(X) is often replaced with popcount(X)
Rank/Select DictionaryCase 2. Sparse (RRR) Case 1. is redundant when m << n A small block X is represented by the class and the offset.
The class C(X) is the number of ones in X e.g . X=101, C(X) = 2.
The offset O(X) is the lexicographical rank among C(X) e.g. X0=011, X1=101, X2 =110, O(X0)=0, O(X1)=1, O(X2)=2
Store a pointer of every (log n)2/2 blocks and an offset for each
element
B = 111110100101010000100111010
C = 3 2 1 2 1 0 1 3 1
O = 0 2 2 1 1 0 2 2 1
Rank/Select Implementation Case 2. Sparse (RRR) Ida: when m << n, many classes (# of ones in small block) are
small, and the offsets are also small
Total size is nH0(B) + o(n) bits. Rank is supported in O(1) time We can also support select in O(1) time using the same working
space.
Note: RRR is also efficient when ones and zeros are clustered. The following bit is not sparse, but can be efficiently stored by
RRR encoding.
000000100000100011111111110111110111111111100000000010000001
many 0s many 1s many 0s
Rank/Select Implementation Case 3. Very Sparse (SD array) Case 2. is not efficient for very sparse case.
o(n) term cannot be negligible when nH0 is very small Let S[i] := select1(B, i) and t = lg (n/m). L[i] := the lowest t bits of S[i] P[i] := S[i] / 2t
Since P is weakly increasing integers, use prefix sums to
represent H Concatenate unary codes of P[i]-P[i-1].
B=0001011000000001
S[0, 3] = [3, 5, 6, 15], t = lg(16 / 4) = 2.
L[0, 3] = [3, 1, 2, 3]
P[i] = [0, 1, 1, 3], H = 1011001
Rank/Select Implementation Case 3. Very Sparse (SDArray) L is stored explicitly, and H is stored with Rank/Select dictionary. Size
H : 2m bits (H[m] ≦ m)L : m log2 (n/m) bits.
select(B, i) = (select1(H, i)-i)2t + L[i] O(1) time
rank(B, i) = linear search from 1 + rank1(H, select0(H, i)) expected O(1) time.
Rank/Select for large alphabets
Consider Rank/Select on a string with many alphabets I will explain only wavelet trees here since it is very very
useful !
I will not discuss other topics of rank / select for large alphabets Multi-array wavelet trees [Ferragina+ ACM Transaction 07] Permutation based approach [Golynski SODA 06] Alphabet partitioning [Barbey+ ISSAC 10]
Wavelet Tree
Wavelet Tree = alphabet tree + bit vectors in each node
T = ahcbedghcfaehcgd
a b c d e f g h
Wavelet Tree
Wavelet Tree = alphabet tree + bit vectors in each node
ahcbedghcfaehcgd 010010110
acbdc hegh
a b c d e f g h
Filter characters according to its value
For each character c := T[i] (i = 0…n), we move c to the left child and store 0if c is the descendant of the left child.We move c to the right child and store 1 otherwise.
Wavelet Tree
Wavelet Tree = alphabet tree + bit vectors in each node Recursively continue this process. Store a alphabet tree and bit vectors of each node
ahcbedghcfaehcgd 0100101101011010
acbdcacd heghfehg 01011011 10110011
aba cdccd efe hghhg010 01001 010 10110
a b c d e f g h
0 1
Wavelet Tree (contd.)
Total bit length is n lg σ bits The height is l2σ, and there are n bits at each depth.
0100101101011010
01011011 10110011
010 01001 010 10110
a b c d e f g n
Height is lg σ
Wavelet Tree (Lookup)
Lookup(T, pos) e.g. Lookup(T, 9)
0100101101011010
01011011 10110011
010 01001 010 10110
a b c d e f g n
0 1
b := B[pos] = 0pos := Rankb(B, pos) = 4
b := B[pos] = 0pos := Rankb(B, pos) = 2
pos := 9
b := B[pos] = 0
Return “c”
Wavelet Tree (Rank)
Rankc (T, pos) e.g. Rankc(T, 9)
0100101101011010
01011011 10110011
010 01001 010 10110
a b c d e f g n
0 1
b := B[9] = 0pos := Rank0(B, 0) = 4
b := B[4] = 4pos := Rank1(B, 4) = 2
b := B[2] = 0pos := Rank0(B, 2) = 1
Return 1
ahcbedghcfaehcgd
Wavelet Tree (Select)
Selectc (T, i) Example Selectc(T, 1)
0100101101011010
01011011 10110011
010 01001 010 10110
a b c d e f g n
0 1
i := Select1(B,2) = 4
i := Select0(B,1) = 2
i=1
i := Select0(B,4) = 8
Wavelet Tree (Quantile List)
Quantile(T, p, sp, ep) : Return the (p+1)-th largest value in T[sp,
ep)
E.g. Quantile(T, 5, 4, 12)0100101101011010
01011011 10110011
010 01001 010 10110
a b c d e f g n
0 1
# of zeros in B[4,12] is 33 < (5+1) and go to right
# of zeros in B[1,6] is 33 > 2 and go to left
# of zeros in B[0, 2] is 22 = 2 and go to right
Return “f”
43782614ahcbedghcfaehcgd
Wavelet Tree is so calleddouble-sorted array
Wavelet Tree (top-k Frequent List )
FreqList (T, k, sp, ep) : Return k most frequent values in T[sp, ep)
E.g. FreqList(T, 5, 4, 12)
0100101101011010
01011011 10110011
010 01001 010 10110
a b c d e f g n
0 1
Greedy Search on the tree.
Larger ranges will be examined first.
Q := priority queue {# of elems, node}Q.push {[ep-sp, root]}while q = Q.pop() If q is leaf, report q and return Q.push([0’s in range, q.left]) Q.push([1’s in range, q.right])end while
Wavelet Tree (RangeFreq, RangeList)
RangeList (T, sp, ep, min, max) : Return {T[i]} s.t. min≦ [i]<max,
sp≦i<ep RangeFreq(T, sp, ep, min, max) : Return |RangeList(T, sp, ep, min,
max)|
E.g. RangeList(T, 4, 12, c, e)01234578901234560100101101011010
01011011 10110011
010 01001 010 10110
a b c d e f g n
0 1
Enumerate nodes that go to betweenmin and max
For RangeList, # of enumerated nodes is O(log σ + occ)
For RangeFreq, # of enumerated nodes is O(log σ)
Wavelet Tree Summary
Input T[0…n), 0≦T[i]<σ
Many operations can be supported in O(log σ) time using wavelet
treeDescription Time
Lookup T[i] O(log σ)
Rankc(T, p) the number of c in T[0…p) O(log σ)
Selectc(T, i) the pos of (i+1)-th c in T O(log σ)
Quantile(T, s, e, r) (r+1)-th largest val in T[s, e) O(log σ)
FreqList(T, k, s, e) k most frequent values in T[s, e)
Expected O(klog σ)Worst O(σ)
RangeFreq(T, s, e, x, y)
items x ≦ T[i] < y in T[s, e) O(log σ)
RangeList(T, s, e, x, y) List T[i] s.t. x ≦ T[i] < y in T[s, e)
O(log σ + occ)
NextValue(T, s, e, x) smallest T[r] > x, s.t. s ≦ r ≦ e O(log σ)
Wavelet tree improvements
Huffman Tree shaped Wavelet Tree (HWT) If the alphabet tree corresponds to the Huffman tree for T, the
size of the wavelet tree becomes nH0 + o(nH0) bits. Expected query time is also O(H0) when the query distribution is
same as in T
Wavelet Tree for large alphabets For large alphabet set (e.g. σ ≒, n) the overhead of alphabet tree,
and bit vectors could large. Concatenate bit vectors of same depth. In traversal of tree, we
iteratively compute the range of the original bit array.
2D Search using Wavelet Tree
Given a 2D data set D={(xi, yi)}, find all points in [sx, sy] x [ex,
ey] By using a wavelet tree, wen can solve this in O(log ymax + occ)
time
0 1 2 3 4 5 6
0 *
1 * *
2
3 * * *
4 *
5 *
6 *
0 1 2 2 4 4 6 6 6
0 *
1 * *
2
3 * * *
4 *
5 *
6 *
Since each column hasone value, we can convertit to the stringT =313613046
Store the x informationusing a bit arrayXb = 101011001100111
Remove columns with no data.If there are more than one data in the same column, we copy the column so thateach column has exactly one data.
2D Search using Wavelet Tree (contd.)
[sx, sy] x [ex, ey] region is found by s’ := Select0(X, sx)+1 , e’ := Select0(X, ex)+1 s’’ := Rank1(X, s’) e’’ := Rank1(X, e’)
Perform RangeFreq(s’’, e’’, sy, ey)
T =313613046
X = 101011001100111
0 1 2 3 4 5 6
0 *
1 * *
2
3 * * *
4 *
5 *
6 *
Full-text Index
Full-text Index
Given a text T[0, n) and a query Q[0, m), support following
operations. occ(T, Q) : Report the number of occurrence of Q in T locate(T, Q) :Report the positions of occurrence of Q in T
Example
i 0123456789A
T abracadabra
occ(T, “abra”)=2
locate(T, “abra”)={0, 7}
Suffix Array
Let Si = T[i, n] be the i-th suffix of T Sort {Si}i=0…n by its lexicographical order Store sorted index in SA[0…n] in n lg n bits (SSA[i] < SSA[i+1])
S0 abracadabra$S1 bracadabra$S2 racadabra$S3 acadabra$S4 cadabra$S5 adabra$S6 dabra$S7 abra$S8 bra$S9 ra$S10 a$S11 $
S11 $S10 a$S7 abra$ S0 abracadabra$S3 acadabra$S5 adabra$S8 bra$ S1 bracadabra$S4 cadabra$S6 dabra$S9 ra$S2 racadabra$
11107035814692
Search using SA
Given a query Q[0…m), find the range [sp, ep) s.t. sp := max{i | Q ≦ SSA[i]}
ep := min{i | Q > SSA[i]} Since {SSA[i]} are sorted, sp and ep
can be found by binary search in O(m log n) time Each string comparison require O(m) time.
The occurrence positions are SA[sp, ep) if ep=sp then Q is not found on T
Occ(T, Q) : O(m log n) time. Locate(T, Q) : O(m log n + Occ) time
S11 $S10 a$S7 abra$ S0 abracadabra$S3 acadabra$S5 adabra$S8 bra$ S1 bracadabra$S4 cadabra$S6 dabra$S9 ra$S2 racadabra$
Q=“ab”, sp=2, ep=4
Suffix Tree
A trie for all suffixes of T Remove all nodes with only one child. Store suffix indexes (SA) at leaves Store depth information at internal nodes, and
edge information is recovered from depth and SA . Fact: # of internal nodes is at most N-1
Proof: Consider inserting a suffix one by one.
At each insertion, at most one internal node will
be build.
Many string operations can be supported by using
a suffix tree.
11
$
a
10$ bra 7
0c
3
5
d
bra8
1
4
6
9
2
c
d
ra
$
c
$
c
$
c
Burrows Wheeler Transform (BWT)
Convert T into BWT as follows
BWT[i] := T[SA[i] – 1] if SA[i] = 0, BWT[i] = T[n]
abracdabra$->ard$rcaaaabb
i SA
BWT
Suffix
0 11 a $
1 10 r a$
2 7 d abra$
3 0 $ abracadabra$
4 3 r acadabra$
5 5 c adabra$
6 8 a bra$
7 1 a bracadabra$
8 4 a cadabra$
9 6 a dabra$
10 9 b ra$
11 2 b racadabra$
Characteristics of BWT
Reversible transformation We can recover T from only BWT
Easily compressed Since similar suffixes tend to have preceding characters, same
characters tend to gather in BWT. bzip2 Implicit suffix information
BWT has same information as suffix arrays, and can be used for
full-text index (so called FM-index).
Example : Before BWT
When Farmer Oak smiled, the corners of his mouth spread till they
were within an unimportant distance of his ears, his eyes were
reduced to chinks, and diver gingwrinkles appeared round them,
extending upon his countenance like the rays in a rudimentary
sketch of the rising sun. His Christian name was Gabriel, and on
working days he was a young man of sound judgment, easy
motions, proper dress, and general good character. On Sundays he
was a man of misty views, rather given to postponing, and
hampered by his best clothes and umbrella : upon the whole, one
who felt himself to occupy morally that vast middle space of
Laodiceanneutrality which lay between the Communion people of
the parish and the drunken section, -- that is, he went to church,
but yawned privately by the time the con+gegation reached the
Nicene …
Example : After BWT
ooooooioororooorooooooooorooorromrrooomooroooooooormoorooor
orioooroormmmmmuuiiiiiIiuuuuuuuiiiUiiiiiioooooooooooorooooiiiioooi
oiiiiiiiiiiioiiiiiieuiiiiiiiiiiiiiiiiiouuuuouuUUuuuuuuooouuiooriiiriirriiiiriiiiiiaii
iiioooooooooooooiiiouioiiiioiiuiiuiiiiiiiiiiiiiiiiiiiiiiioiiiiioiuiiiiiiiiiiiiioiiiiiiiiiiiiio
iiiiiiuiiiioiiiiiiiiiiiioiiiiiiiiiioiiiioiiiiiiioiiiaiiiiiiiiiiiiiiiiioiiiiiioiiiiiiiiiiiiiiiuiiiiiiiiiiiiiiiiii
oiiiiiiiioiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiuuuiioiiiiiuiiiiiiiiiiiiiiiiiiiiiiiioiiiiuioiuiiiiiiioiiiiiiiu
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiioaoiiiiioioiiiiiiiioooiiiiiooioiiioiiiiiouiiiiiiiiiiiiooiiiiiiiiiiiiiiiiii
iiiiiiiiiiiiioiiiiiiiiiiiiiiiiiiioiooiiiiiiiiiiioiiiiiuiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiioiiiiiiiiiiiiioiiiuiii
iiiiiiioiiiiiiiiiiiiuoiiioiiioiiiiiiiiiiiiiiiiiiiiiiuiiiiuuiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiuiuiiiiiuuiiiii
iiiiiiiiiiiiiiiiiiiuiiiiiiiiiiiiiiiiiiiiiiiiiiiioiiiiiiioiiiiiiiiiiiiiiiiiiiiioiiiiiiiiioiiiiuiiiioiiiioiiiiiiiiii
iiiiiiiiiiiiiiiiiiiiiiiiiiiioiioiiiiiiuiiiiiiiiiiiiiiiooiiiiiiiiiiiiiiiiiiiioooiiiiiiiioiiiiouiiiiiiiiiiiiiiii
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiioiiioiiiiiiioiiiiiioiiiiuiuoiiiiiiiiioiiiiiiioiiiiiiiiiiiiiiiiiiiiiiiiiiiioi
oi….
LF-mapping
Let F[i] = T[SA[i]] First characters of suffixes
Fact : the order of same chars
in BWT and F is same
i BWT Suffix (First char is F)
0 a1 $
1 r a1$
2 d a2bra$
3 $ a3bracadabra$
4 r a4cadabra$
5 c a5dabra$
6 a2 bra$
7 a3 bracadabra$
8 a4 cadabra$
9 a5 dabra$
10 b ra$
11 b racadabra$
LF-mapping (1/3)
Let F[i] = T[SA[i]] First characters of suffixes
Fact : the order of same chars
in BWT and F is same
Proof: the order of chars in BWT
is determined by the following
suffixes.
the order of chars in F is also
determined by the following
suffixes. These are same !
i BWT Suffix (First char is F)
0 a1
1 r a1
2 d a2
3 $ a3bracadabra$
4 r a4
5 c a5
6 a2
7 a3 bracadabra$
8 a4
9 a5
10
11
Same
LF-mapping (2/3)
Let F[i] = T[SA[i]] First characters of sorted suffixes
Let C[c] be the number of
characters smaller than c in T C[c] = beginning position of F[i] = c
For B[i]=c, its corresponding char in F[i] is found by Rankc(T, i) + C[c]
i BWT Suffix (First char is F)
0 a $
1 r a$
2 d abra$
3 $ abracadabra$
4 r acadabra$
5 c adabra$
6 a bra$
7 a bracadabra$
8 a cadabra$
9 a dabra$
10 b ra$
11 b racadabra$
LF-mapping (3/3)
Let ISA[0, n) s.t. ISA[r]=p if SA[p] = r SA[i] converts the rank into the position, and ISA[i] vice versa.
For c := BWT[r], we define two functions LF(r) := Rankc(T, r) + C[c] = ISA[SA[r] -1]
PSI(r) := Selectc(T, r– C[c]) = ISA[SA[r] + 1] LF and PSI are inverse functions each other.
Using LF and PSI, we can move a text in its position order. PSI = forward by one char , LF = backward by one char
e.g. T = abracadabra
SA[r] SA[PSI[r]]SA[LF[r]]
Decoding of BWT using LF-mapping
r := beg // the rank of T[0, n), i.e. ISA[0]
while
Output BWT[r]
r := PSI(r)
until r != beg
That’s all !
In practice, PSI values are pre-computed using O(n) time
Compressed Full-text Index
Compressed Full-Text Index
Full-text can be compressed Moreover, we can recover original text from compressed index
(self-index)
I explain FM-index only here.
Other important compressed full-text indices Compressed Suffix Arrays LZ-index LZ77-based
FM-index
Using LF-mapping for searching
pattern.
Assume that a range [sp, ep)
correspond to a pattern P.
Then if a pattern P’ = cP occurs,
then [sp, ep) should be preceded by c
i BWT Suffix
0 a $
1 r a$
2 d abra$
3 $ abracadabra$
4 r acadabra$
5 c adabra$
6 a bra$
7 a bracadabra$
8 a cadabra$
9 a dabra$
10 b ra$
11 b racadabra$
P=a appeared in [1, 5)
P’=ra appear in P
FM-index
Using LF-mapping for searching
pattern.
Assume that a range [sp, ep)
correspond to a pattern P.
Then if a pattern P’ = cP occurs,
then [sp, ep) is preceded by c
The corresponding suffixes can befound by Rankc(BWT, sp) + C[c]
and Rankc(BWT, ep) + C[c]
i BWT Suffix
0 a $
1 r1 a$
2 d abra$
3 $ abracadabra$
4 r2 acadabra$
5 c adabra$
6 a bra$
7 a bracadabra$
8 a cadabra$
9 a dabra$
10 b r1a$
11 b r2acadabra$
P=a appeared in [1, 5)
P’=ra appear in P
FM-index
Input query : Q[0, m)
Output a range [sp, ep) corresponding to Q
sp := 0, ep := n
for i = m-1 to 0
c := Q[c] sp := C[c] + Rankc(BWT, sp)
ep := C[c] + Rankc(BWT, ep)
end for
Return (sp, ep)
Occ(T, Q) = ep - sp
Locate(T, Q) = SA[sp, ep)
Storing Sampled SA’s
Store sampled SAs instead of original SA. Sample SA[p] s.t. SA[p] = tk for (t=0…) We also store a bit array R to indicate the index of
sampled SAs.
Recall that LF(r) = ISA[SA[r]-1] for c = BWT[c]. If SA[p] is not sampled, we use LF to move on SA
SA[r] = SA[LF(r)]-1 = SA[LF2(r)]-2 = … = SA[LFk(r)]-k We move on SA until the sampled point is found
using at most k LF operations.
Set k = log1+εn. The size of A is (n/k) log n = o(n) The lookup of SA[i] requires O(log1+εn logσ) time.
i SA R
0 0
1 0
2 0
3 0 1
4 0
5 0
6 8 1
7 0
8 4 1
9 0
10 0
11 0
Grammar Compression on SA [R. Gonzalez+ CPM07] SA can also be compressed using grammar based compressions Let DSA[i] := SA[i] – SA[i-1] (for i > 0) If BWT[i] = BWT[i+1] = … = BWT[i+run] then
DSA[i+1, i+run] = DSA[k+1, k+run] for k := LF[i] DSA contains many repetition if BWT has many runs.
DSA can be compressed using grammar compression [R. Gonzelez+ CPM07] used Re-pair but other grammar
compressions can be used as long as they support fast
decompression.
Compression Boosting in FM-index
The size of FM-index using wavelet trees is nH0 + o(nH0) bits We can improve the space to nHK by compression boosting Compression Boosting [P. Ferragina+ ACM T 07]
Divide BWT into blocks using contexts of length k Build a separate wavelet tree for each block
Implicit Compression Boosting [Makinen+ SPIRE 07] Wavelet Tree + compressed bit vectors using RRR in Wavelet Tree
The size is close to optimal: the overhead is at most σk+1 log n, which is o(n) when k ≦ ((1-ε)logσn)-1
bbabdd
aaaa..aabb..aacc..aacd..abcd..abcc..
Divide BWT’susing followingsuffix of length 2.
Compression Boosting using fixed length block[Karkkaine+ SPIRE 11] Dividing BWT into blocks of fixed length can also achieve nHK
Block size is σ(log n)2
To analyze the working space, we calculate the overhead incurred
of conversion from context-sensitive partition into fixed-length
partition
Note. This encoding is very similar to the encoding used in bzip2
SSA: Huffman WTAFFMI : Context Sensitive Partition of BWTFixed Block : Fixed Block
+ RRR : Represent bit vectorsin WT using RRR encoding
Summary of state-of-the-art
Give an input String T, compute BWT for T (=B). Divide B into blocks of fixed length Each block is stored in a Huffman-shaped wavelet tree. A bit-vector in a wavelet tree is stored using RRR representation. For English, 2-3 bits per character, 1 pattern 0.06 msec.
Analyzing Document Collection
Document Query
DL(D, P) : Document Listing List the distinct documents where P appears
DLF(D, P) :Document Listing with Frequency The result of DL(D, P) + frequency of P in each document
Top(k, D, P) : Top-k Retrieval List the k documents where P appears with largest scores S(D, P) e.g. S(D, P) : term frequency, document’s score (e.g. Page rank)
In the previous section we consider hit positions, and here we
consider hit documents We need to convert hit positions into hit documents. This is not easy.
Index for document collection
For a document collection D={d1, d2, …, dD} Build T = d1#d2#d3#...#dD and build an full-text index for T
We also use a bit array to indicate the beginning positions of docs. Given a hit position in T, we can convert it to the hit doc in O(1)
time.
Baseline of DL and DLF Perform Locate(T, Q) and convert each hit positions into a hit doc
each independently. However, this requires O(occ(T, Q)) time, and occ(T, Q) could be
much larger than # of hit documents
DocArray and PrevArray (1/2)[Muthukrishnan SODA 02]
DocArray : D[0, n] s.t. D[i] is the doc ID of SA[i]. DL(D, Q) is now to list all distinct values in the hit range D[sp, ep)
PrevArray : P[0, n] s.t. P[i]=max{j < I, D[j]=D[i]} P[i] can be simulated as selectD[i] (D, rankD[i](D, i-1))
i 0 1 2 3 4 5 6
D 66 77 66 88 77 77 88
P -1 -1 0 -1 1 4 3
DocArray and PrevArray (2/2)
Alg. DocList reports all distinct doc IDs in [sp, ep) in O(|DL|) time. Call DocList(sp, sp, ep)
i 0 1 2 3 4 5 6
D 66 77 66 88 77 88 88
P -1 -1 0 -1 1 3 5
RMQ(P, 2, 7) = -1 (=P[3])Report D[3]
RMQ(P, 2, 3) = 0 (=P[2])Report D[2]
RMQ(P, 5, 7) = 3 (=P[5])Return
DocList(gsp, sp, ep) IF sp=ep RETURN; (min, pos) := RMQ(P, sp, ep) IF min ≧ gsp RETURN; Report D[pos] DocList(gsp, sp, pos) DocList(gsp, pos+1, ep)
Wavelet Tree + Grammar Compression[Navarro+ SEA 11]
Build a wavelet tree for D. Using Freqlist(k, sp, ep), we can solve DL and DLF in O(log σ)
time. However storing D requires n lg |D| bits. This is very large.
As in compression of suffix arrays using grammar compression, D
can also be compressed by grammar compression Apply a grammar compression to bit arrays in wavelet trees, and
support rank/select operations
Currently, this is the fastest/smallest approach in practice.
Suffix Tree based approach (1/4)[W.K. Hon+ FOCS 09] Let ST be the generalized suffix tree for a collection of
documents A leaf l is marked with document d if l corresponds to d An internal node l is marked with d if at least two children of v
contain leaves marked with d An internal node can be marked with many values d.
a c b a a c c b c
3 710
2
1
4 5 6 8 911
12
13
14
a c
a b c
c
Suffix Tree based approach (2/4)
In every node v marked with d, we store a pointer ptr(v, d) to its
lowest ancestor u such that u is also marked with d ptr(v, d) also has a weight corresponds to relevance score. If such ancestor does not exist, ptr(v, d) points to a super root
node.
a c b a a c c b c
3 710
2
1
4 5 6 8 911
12
13
14
a c
a b c
cptr(7, a) = 2
ptr(13, a) = 2
Suffix Tree Based Approach (3/4)
Lemma 1. The total number of pointers ptr(*,*) in T is at most 2n Lemma 2. Assume that document d contains a pattern P and v is
the node corresponding to P. Then there exist a unique pointer
ptr(u, d), such that u is in the subtree of v and ptr(u, d) points to
an ancestor of v
c c b
10
2
1
11
12
13
c
a b c
c
Example: when v = 10. ptr(10, c)=2, and ptr(13, b)=2.These ptrs correspond to the occurrence of P in b and c.
Suffix Tree Based Approach (4/4)
To enumerate document corresponding to the node x, find all
ptrs that begins in the subtree of x, and points to the ancestor of
x. These ptrs uniquely corresponds to each hit documents (Lemma)
x
Geometric Interpretation
a c b a a c c b c
3 710
2
1
4 5 6 8 911
12
13
14
a c
a b c
c
-1012
1 2 2 2 4 5 6 7 8 9 10 11 12 13 14
c a bc
a c b a
a a
c
c c
b
c
depth
0
1
2
3
pd(v, c) := depth of ptr(v, c)
place all ptrs ina point (x, pd(v,c))
pd
v=2v=10
Document Query = three-sided 2D query
When v := locus(Q) d := depth(v), and r be the maximal index
assigned to v or its descendants. DL can be solve by
RangeList(v, r, 0, d-1)
-1012
1 2 2 2 4 5 6 7 8 9 10 11 12 13 14
c a bc
a c b a
a a
c
c c
b
c
pd
Top-K query
Each ptr(v, d) can store any score function depended on Q, D. For each node v, f ptr(x, d) points to vTop-K
Theorem [Navarro+ SODA 12] k most highly weighted points in
the range [a, b] x [0, h] can be reported in decreasing order of
their weights in O(h+k) time
In practice, we can use a priority queue on each node.
Bibliography
[Navarro+ SEA 12] Practical Compressed Document Retrieval, G.
Navarro, S. J. Puglisi, and D. Valenzuela, SEA 2012
[Navarro+ SODA 12] Top-k Document Retrieval in Optimal Time and
Linear Space, SODA 2012
[T. Gagie+ TCS+ 11] New Algorithms on Wavelet Trees and
Applications to Information Retrieval, TCS 2011
[G. Navarro+ 11] Practical Top-K Document Retrieval in Reduced
Space
[Karkkaine+ SPIRE 11] Fixed Block Compression Boosting in FM-
indexes, J. Karkkainen, S. J. Puglisi, SPIRE 2011
[W.K. Hon+ FOCS 09] Space-Efficient Framework for Top-k String
Retrieval Problems, W. K. Hon, R. Shah, J. S. Vitter, FOCS 2009
[Navarro+ SEA 11] “Practical Compressed Document Retrieval”,
Gonzalo Navarro , Simon J. Puglisi, and Daniel Valenzuela,, SEA
2011
[Navarro+ SODA 12] Top-k Document Retrieval in Optimal Time and
Linear Space , Gonzalo Navarro, Yakov Nekrich, SODA 2012
[Gagie+ JTCS 11] New Algorithms on Wavelet Trees and Applications
to Information Retrieval, Travis Gagie , Gonzalo Navarro , Simon J.
Puglisi, J. of TCS 2011