getting started with unstructured data

November 17, 2011

Getting Started with Unstructured DataChristine Connors & Kevin LynchTriviumRLG LLC

Thursday, November 17, 2011

Meta

✤ Presenter: Christine Connors

✤ @cjmconnors

✤ Presenter: Kevin Lynch

✤ @kevinjohnlynch

✤ Principals at www.triviumrlg.com

✤ Partnering with Dataversity


http://www.triviumrlg.com


Agenda

✤ What is unstructured data?

✤ Where do we find it?

✤ How important is it?

✤ How do we visualize it?

✤ Machine processing for actionable data

✤ Tools


What is unstructured data?

✤ Data which is

✤ Not in a database

✤ Does not adhere to a formal data model

✤ Content


Isn’t that a misnomer?

✤ Problematic term

✤ The presence of object metadata or aesthetic markup does not alone give ‘structure’ in this sense of the word

✤ Object metadata = machine or applied properties

✤ Aesthetic markup = stylesheets; rendering information

✤ Semi-structured data is typically treated as unstructured for the purposes of machine processing and analysis


Types of ‘un’structured data

✤ Text-based documents

✤ Word processing, presentations, email, blogs, wikis, tweets, web pages, web components (read/write web)

✤ Audio/video files


Where do we find it?

✤ Office productivity suites

✤ Content management systems

✤ Digital asset management systems

✤ Web content management systems

✤ Wikis, blogs, comment & discussion threads

✤ Social networking tools

✤ Twitter, Yammer, instant messengers


85%

15%

Structured Unstructured

Is it really that important?


What’s in that 80-85%?

✤ Progress reports - created in a word processor



✤ Dashboards - created in presentation software



✤ Progress reports - color coded text in a spreadsheet



✤ Brainstorming - in messaging systems

✤ Decision making - in email



✤ Business intelligence - on the web and more


How can we make the data more actionable?

✤ Identify it

✤ Convert to a format you can work with

✤ Add structure, meaning:

✤ information extraction

✤ annotation

✤ content analytics


What about enterprise search?

✤ First line of defense

✤ Points you at the highest relevancy ranked data via pattern matching and statistical analysis

✤ Does not assist in other visualizations or transformations without further machine processing


Information Extraction

✤ Token identification - “tokenization”

✤ Part-of-speech tagging - “POS” tagging (noun, verb, adverb, adjective, etc.)

✤ Phrase identification - noun phrase

✤ Entity extraction - people, places, events, dates, organizations


Information Extraction

✤ Cluster analysis - group related information, where relationship may not be known

✤ Classification - mapping to specific categories

✤ Dependency identification / Rule generation

✤ Relationship detection - e.g. “Joe” “is CEO” at “IBM”

✤ Summarization - key concepts or key sentences


Open Tools

✤ GATE – General Architecture for Text Engineering, from the University of Sheffield, with many users and excellent documentation.

✤ GATE has customizable document and corpus processing pipelines. GATE is an architecture, a framework, and a development environment, with a clean separation of algorithms, data, and visualization.


Open Tools

✤ UIMA – Unstructured Information Management Architecture (IBM’s Watson uses this), originated at IBM, now an Apache project.

✤ Component software architecture with a document processing pipeline similar to GATE. Focus on performance and scalability, with distributed processing (web services).


UIMA

Fred is theCenter CEO of

OrganizationPerson

CeoOf

Arg2:OrgArg1:Person

PPVPNPParser

Named Entity

Relationship

Center Micros

Common Analysis Structure (CAS)

Artifact (e.g., Document)

Analysis Results (i.e., Artifact Metadata)

UIMA CASRepresentation now

Alignedwith XMI standard

UIMA’s Basic Building Blocks are Annotators. They iterate over an artifact to discover new types based on existing ones and update the Common Analysis Structure (CAS) for

upstream processing.

Chart byIBM


UIMA

Image byIBM


Commercial Tools

✤ Oracle Data Mining (Text Mining)

✤ IBM SPSS

✤ SAS Text Miner

✤ Smartlogic

✤ Lots of acquisitions going on in the “big data” space

✤ HP acquired Autonomy

✤ Oracle acquired Endeca


A Note on Tools

✤ UIMA and GATE – comprehensive suite of capabilities, with learning curves.

✤ Commercial tools range from unstructured capabilities inside DBMSs like Oracle, to Business Objects business intelligence tools (who acquired Inxight from Xeroc Parc).

✤ Your mileage will vary. The biggest differentiator is your knowledge of your data.


What can unstructured data look like post-processing?


Machine Processing

Machine Processing Platform

Natural Language Processing

Statistical Analysis

Rules-based Classifica-

tion

Semantic Analysis

Unstructured Data

IndexAPI

Visualizations

Federated Search

Data StoresThursday, November 17, 2011

Questions?


Thank youChristine ConnorsKevin Lynchwww.triviumrlg.com




getting started with unstructured data

Education