Textové Databázy

Ján GENČIPDT

2

Obsah

• Literatúra• Terminológia• Vymedzenie pojmu textové databázy• Typy dotazov• Fulltextové vyhľadávanie• Lingvistické korpusy

3

Literatúra• Pokorný J.: Databázové systémy 2, Nakladatelství

ČVUT, 2007• Pokorný J., Snášel V., Kopecký M.: Dokumentografické

informačné systémy, Nakladatelství Karolinum, 2005.• Laura C. Rivero, Jorge H. Doorn, Viviana E. Ferraggine:

Encyclopedia Of Database Technologies And Applications. Idea Group Publishing, 2005 (heslo Text Databases, p. 688)

• Erickson J.: Database Technologies: Concepts, Methodologies, Tools, and Applications. IGI Global, 2009. ISBN 978-1-60566-058-5 (pp. 931-939)

4

Literatúra (cont.-1)

5

Literatúra (cont.-2)

• Oracle Text. http://www.oracle.com/technology/products/text/index.html

• Oracle Text. An Oracle Technical White Paper. June, 2007 (prečítať) http://www.oracle.com/technology/products/text/pdf/11goracletexttwp.pdf

6

TXT DB - Terminológia

• Textové databázy (informačné systémy)• Dokumentové databázy (Document

databases)• Dokumentografické informačné systémy

7

Text Databases – definition• A text is any sequence of symbols (or characters) drawn from an

alphabet.• A large portion of the information available worldwide in electronic

form is actually in text form (other popular forms are structured and multimedia information):– natural language text (e.g., books, journals, newspapers, jurisprudence

databases, corporate information, the Web), – biological sequences (e.g., ADN and protein sequences),– continuous signals (e.g., audio and video sequence descriptions, time

functions), – and so on.

• A text database is a system that maintains a (usually large) text collection and provides fast and accurate access to it. These two goals are relatively orthogonal, and both are critical if one is to profit from the text collection.

8

TXT DB - Type of queries– Syntactic search (expressed in the sequence of characters

preseted in the text):• String matching (the simplest query, cely rad algoritmov – Knut-

Morris-Pratt first O(n))• Regular expression• Approximate searching (to recover from different kinds of errors that

the text collection (or the user query) may contain - simple error model is edit distance)

– Semantic search (great value) - user expresses an information need and the system retrieves portions of the text collection (i.e., documents) that are relevant to that need, even if the query words do not directly appear in the answer. System ranks the documents and offers the highest ranked documents to the user. There are no right or wrong answers, just better and worse ones.

9

Fulltext search

• In the traditional database management systems (DBMS), text manipulation is restricted to the usual string manipulation facilities (the exact matching of substrings)

• The traditional string-level operations are very costly for large documents - traditional DBMS engine is inefficient for these operations, they are usually extended with a special full-text search (FTS) engine module.

10

Fulltext search (cont.)

• There is a significant demand on the market on the usage of free text and text mining operations, since information is often stored as free text (text analysis in medical systems, analysis of customer feedbacks, and bibliographic databases)

• Simple character-level string matching would retrieve only a fraction of related documents

11

Alternatives for implementingan FTS engine

• Built-in FTS engine module (Oracle, Microsoft SQLServer, Postgres, and mySQL; Informix Text Datablade; )

• DBMS-independent FTS engine (SPSS LexiQuest, SAS Text Miner, dtSearch, and Statistica Text Miner)

12

Ways of processing

• Text mining• Full text search

13

Text mining• Subfield of document management that aims at

processing, searching, and analyzing text documents

• The goal – to discover the non-trivial or hidden characteristics of individual documents or document collections

• Interdisciplinary field of machine learning which exploits tools and resources from computational linguistics, natural language processing, information retrieval, and data mining

14

General application schema of text mining

15

Information Extraction• Includes i.e. following subtasks:

– named entity recognition – recognition of specified types of entities in free text,

– co-reference resolution – identification of text fragments referring to the same entity,

– identification of roles and their relations – determination of roles defined in event templates

16

Text Categorization• Aim - sorting documents into a given

category system; e.g..:– document filtering –spam filtering, or

newsfeed;– patent document routing – determination of

experts in the given fields;– assisted categorization – helping domain

experts in manual categorization with valuable suggestion;

– automatic metadata generation.

17

Document Clustering• Groupping elements of a document collection

based on their similarity. • Documents are usually clustered based on their

content.• Document Clustering is applied for e.g.:

– clustering the results of (internet) search for helping users in locating information,

– improving the speed of vector space based information retrieval,

– providing a navigation tool when browsing a document collection.

18

Summarization• Automatic generation of short and

comprehensible summaries of documents

19

FULL-TEXT SEARCH (FTS)ENGINES

20

Fulltext indices

• A crucial sub-problem in the information retrieval area is the design and implementation of efficient data structures and algorithms for indexing and searching information objects that are vaguely described.

• The most commonly used indexing structures are:– inverted files, – signature files,– bitmaps.

21

Informix

• Excalibur Text DataBlade Module provides text search capabilities that include:– phrase matching, – exact and fuzzy searches, – compensation for misspelling, – synonym matching.

22

Lingvistické korpusy

• Kolekcie textov v konkrétnom jazyku určené primárne pre lingvistický výskum

• Značkované texty• Príklady:

– British National Corpus (100 mil. slov)– Slovenský národný korpus (530 mil. tokenov)– Český národný korpus (300 mil. slov)

• Paralelné korpusy