data discovery and reuse

Data discovery and reuse

TTF-IP Workshop, 18.3.

Monday, March 21, 2011

Data processes 1

•Need: machine-readable, openly licensed.

•Re-publish derived data


Data processes 2

•Goal: reproducible results, ecosystems:

•Tools to regularly extract data, share, transform and load it.

•Catalogues, documentation.

•“Data-the-process”, not “data-the-file”

•no “Excel Afternoons”


Data Acquisition

VoluntaryRelease

InvolunatryRelease

Active acquisition FoI Scraping

Passiveacquisition PSI/Open Data Leaks


Basic tools

•Language “convention”: Python

•Screen scraping: ScraperWiki

•Semi-structured storage: MongoDB

•Keeping an overview: CKAN


Textual data

•De-PDF (Acrobat Pro, pdf2text)

•Index & Search (Apache Solr)

•Basic NLP: Word counts/freqs, NEE etc.: nltk

•Publish: Co-ment, AnnotateIt, Scribd

•Soon: DocumentClouds for all!


Numeric data I•De-PDF: ABBYY FineReader

•Munge & Massage: Google Refine

•Share & Extend: Google Spreadsheets

•R/Stata/SPSS: more suited to internal processes.

•BI/Analytics/Aggregation: custom?


Numeric data II

•Visualization, first go: Google Vis Toolkit, IBM ManyEyes

•Visualization, interactive: Protovis, Raphael

•Flash considered harmful :-(


Network data

•Can be derived from other types

•Think about structure: nodes, edges, weights, directions

•Analysis: find central actors, mediators, ...: MCI, NetworkX

•Visualization: Gephi, GraphViz


FTS “Afghanistan”, Ronny PatzMonday, March 21, 2011

Geo-data

•There is more than Google Maps markers :-)

•Talk to your neighborhood OSM crowd.


data discovery and reuse

Documents