Global Sensor NetworksAn evolving middleware for sensor data

stream processing

Jean-Paul Calbimonte, Julien Eberle, Sofiane Sarni, Karl Aberer

LSIR EPFLMountain Observatories 2014, Reno Nevada

16.07.2014

http://gsn.epfl.ch

Sensor deployments everywhere

MountainsGlaciersSnow regionsSea CoastalAgriculture…

DIYMobileParticipatory

We want the data

• Open data repositories• Accessible research datasets• Discoverability• Reuse datasets• Metadata

GSN: Global Sensor Networks

GSN: Global Sensor Networks

Help managing sensor datasets

Help publishing the data

Help making the data discoverable and reusable

GSN in a nutshell

• Middleware: Sensor network deployment

• Virtual Sensor (VS): Process streaming data

• Hosts & manages multiple VSs

6

Where is GSN?

7

• Sensor Network:• Sensing• In network

• Data/query Processing• Filtering• Aggregation

• Data Management:• Data:

• Management.• Publishing.• Expensive processing.• Archives.

DataManagement

SensorNetwork

GSN goes here

Collecting data from different sources

8

• GSN works in a distributed fashion• Data can be kept locally• Break data silos

• Put sensor data on the webGSN nodes

GSN Distributed Deployment

9

Integrity Service

Access ControlGSN/Web/Web-Services

Notification ManagerQuery ProcessorQuery RepositoryStorage Manager

Virtual Sensor ManagerInput Stream Manager

Stream Quality ManagerLife Cycle Manager

Pool Of Sensing Devices

GSN Virtual Sensors

10

• A virtual sensor, any kind of data producer• a real sensor, a wireless camera, a desktop computer, GPS sensor, network traffic,

etc.• combination of other virtual sensors.• Logical view of the sensor network.• Described in an XML file:

• Functional/non-function properties.

Source 1

Source 2…

Source n

Application logicand processing

Output Stream

Virtual S

ensor

Virtual Sensor configuration

11

<virtual-sensor name="room-monitor" > <addressing> <predicate key="geographical">BC143</predicate> <predicate key="usage">

room monitoring</predicate> </addressing> <life-cycle pool-size="10" /> <output-structure> <field name="image" type="binary:image/jpeg" /> <field name="temp" type="int" /> </output-structure> <storage permanent="true" history-size="10h" /> <input-streams> <input-stream name="cam"> <stream-source alias="cam" storage-size="1“

sampling-rate=“1”> <address wrapper=“tinyos2.x"> <predicate key=“host">tinybox.epfl.ch </predicate> <predicate key=“port">9001</predicate> </address> select * from WRAPPER </stream-source><stream-source alias="temperature1“ storage-size="1m“ sampling-rate=“1”>

<address wrapper="remote"> <predicate key="type">temperature</predicate> <predicate key="geographical">BC143-N

</predicate> </address> select AVG(temp1) as T1 from WRAPPER </stream-source> <stream-source alias="temperature2“ storage-size="1m“> <address wrapper="remote"> <predicate key="type“>temperature</predicate>

<predicate key="geographical“>BC143-S

</predicate> </address> select AVG(temp2) as T2 from WRAPPER </stream-source> <query> select cam.picture as image, temperature.T1 as temp from cam, temperature1 where temperature1.T1 > 30 AND temperature1.T1 = temperature2.T2 </query> </input-stream> </input-streams></virtual-sensor>

Data in GSN through Wrappers

12

Common abstractions, independent of applications, hardware

Simple integration & data correlation.

5140

GS

N

Various Applications

Plug & Play deployment

On-the-fly reconfiguration

GS

N

GSN

Some available mappings

13

• HTTP generic wrapper• devices accessible via HTTP GET or POST requests, e.g., the AXIS206W wireless camera

• Serial forwarder wrapper• enables interaction with TinyOS compatible motes (standard access in TinyOS)

• USB camera wrapper• local USB connection.• supports cameras with OV518 and OV511 chips.

• RFID wrapper• access to Texas Instruments Series 6000 S6700 multi-protocol RFID readers• Alien Technologies long range RFID reader 8950 EU.

• WiseNode wrapper• access to WiseNode sensors (CSEM, Switzerland, http://www.csem.ch/)

• Generic UDP wrapper• any device using the UDP protocol

• Generic serial/bluetooth wrapper• supports sensing devices which send data through the serial port, e.g., EPuck robots, etc.

Wrappers: Lines of Code

14

50RFID reader (TI)

50Generic HTTP

300Wired camera

180Generic serial

45Generic UDP

75WiseNode

160TinyOS

Lines of codeWrapper type

Open source project: Available in Github

• Open Source License• Mainly in Java• Community Support• Used in several projects

Releases available in Github

So what can I do with it?

• Get data from my sensors (API, web interface) • Store and archive the data• Put it online, available for download• Put it online, available for discovery and querying• Apply post-processing to the data• Combine different data sources• Use the data from an R script• More stuff…

Data Validation through Measurements and Modelling over Multiple Scales

LagrangianDispersion Model

High resolution urbanatmospheric pollution maps

Model InputTerrain, meteorology, source strength, background

Sensor DataCrowd-sensors, mobile sensors, monitoring stations

GSN Storage

19

• Centralized RDBMS

Trends: Data & Users

Evaluate a NoSQL solutionScalability Fault-tolerance Performance

GSN Storage Extension

20

LSIR-Cloud

HBASE Java API Client

Experimental Platform

CPU: 24 cores x 2.3GHzMem: 64 GBDisk: 2.8 TB

Nodes: 8CPU: 8 x 12 cores x 2.3GHzMem: 32 GBDFS Disk: 43 TBNetwork: 1 Gbps

HDFS Cluster

Put / Get

HBASEExporter

HBASE Wrapper

HBASE Query Handler

User Requests

VS data

Store VS data

Read VS data

Execute Query

SSN Ontology with other ontologies

21

W3C SSN Ontology

tool for modeling our sensor datacombine with domain ontologies

GSN Access Control (AC)

• VS has an owner: decides user access

22

VS: Virtual Sensor

AC ISSUES REASON

Private VS Features not visible VS Availability should be provided

No Notifications Faster responses, if notified

No Access Time Limitations Enable owner to control access

Manual VS management Automation of the VS activation

No AC in REST services Enable alternative data access

More things we’re doing

• Integration: integrate with Geo-enabled repositories (e.g. GeoNetworks)

• Standards: NetCDF, OGC standards, OpenDAP• Metadata: add semantics to the data• Web standards: RDF and Linked Data• tinyGSN: for mobile devices

Some example of Use

• OpenIoT: Smart agriculture, manufacturing, etc.• SwissExperiment: environmental sensing• PlanetData: traffic data observation• OpenSense: air quality measurements• Permasense: mountain and snow observatory• Etc..

OSPER - Swiss ExperimentOpen support platform for environmental research

Multidisciplinary research team• Real world data + problems

Facilitating research in:• Precipitation patterns in mountains• Evaporation in Africa• Return periods of Natural Hazards• Stream flows in Alpine catchments• Permafrost in the Alps

managing environmental sensor data &metadataPlatform

http://swiss-experiment.chD

ata

heterogeneous sensing devices

summarization, filtering, compression, interpolation

continuous processing, streaming, geospatial, aggregation

pattern discovery, correlation, regression

metadata management, semantics

data services, visualization, standards

acquisition

processing

querying

analysis

discovery

provision

OpenSense2

global concernhighly location-dependenttime-dependent

Crowdsourcing High-Resolution Air Quality SensingAir Pollution

Accurate location-dependent and real-time information on air pollution is needed

Integrated air quality measurement platform

- Heterogeneous devices and data- Human activity assessment, lifestyle and health data

• Link high-quality and low-quality data • Integration of pure statistical models and physical

dispersion models• Better coverage through crowdsensing• Incentives for crowd data provision• Finer temporal and spatial resolutions• Utilitarian approach for trade-off between model

complexity, privacy and accuracy• Higher accuracy of pollution maps models

http://opensense.epfl.ch

Institutional stationsOpenSense infrastructure

Personal mobile sensors CrowdSense

OpenIoT FP7Open Source Cloud solution for the Internet of Things

http://openiot.eu

Established Open-source platform for IoT

• Integrate sensors & things with cloud computing • Configure, deploy and use IoT services• Auditing/assessing privacy of IoT apps in the cloud• Semantic annotations of internet-connected objects• Energy-efficient data harvesting• Publish/subscribe for continuous processing and

sensor data filtering• Mobility of sensors and QoS aspects in IoT

https://github.com/OpenIotOrg/openiot

Use cases and validation scenariosSmart

Manufacturing Campus Guide Air Monitoring Agriculture Sensing

Thanks a lot!

Global Sensor Networks

Jean-Paul Calbimonte

LSIR EPFL

http://gsn.epfl.ch