22 - csiro - water data management-sep-17

Water data management platforms Modern Tools & Techniques for Water Resources Assessments & Management

Amit Parashar

17 September 2014 New Delhi

LAND & WATER FLAGSHIP

Contents

• CSIRO

• Data management challenges

• Data sharing platforms

• Hydrological geofabric

• Cloud computing

• Demo

Top 1% of global research

institutions in 14 of 22 research

fields Top 0.1% in 4 research fields

Darwin

Alice Springs

Geraldton 2 sites

Atherton

Townsville 2 sites

Rockhampton

Toowoomba

Gatton

Myall Vale Narrabri

Mopra

Parkes

Griffith

Belmont

Geelong

Hobart Sandy Bay

Wodonga

Newcastle

Armidale 2 sites

Perth 3 sites

Adelaide 2 sites Sydney 5 sites

Canberra 7 sites

Murchison

Cairns

Irymple

Melbourne 5 sites

CSIRO: Who we are

Werribee 2 sites

Brisbane 6 sites

Bribie Island

People

Locations

Flagships

Budget

6000

58

9

$1B+

Challenges in data management

• Lots of investment in water data collection (collecting, finding, accessing and formatting of data)

• Multiple agencies collecting data – different ways of managing data

• No single point of truth (overlaps between agencies)

• Fragmented data silos

• Difficult to aggregate quality assured data set as input to research and inform policy

Challenges in data management

• Supporting information/data requests again and again and again (usually for the same data) Need scaleable mechanism to share data

• Water management pressures require increasing complex and integrated assessments to support decision makers.

• As India moves towards IWRM & Basin Level Planning, water data managers will increasingly need to support a variety of jurisdictions, agriculture, urban, environment, energy

• Community pressure for access to data (increased transparency)

Discover Access Understand Extract, Transform, Load

Use

Time and effort

Research Focus To enable more efficient and effective management of water by improving the availability, accessibility and usability of existing and new water information products and services.

Action

Knowledge

Information

Data

Action

Knowledge

Information

Data

Spatial Information Service Stack – Data Sharing Platforms

Discovery/Access/Integrate

Why? Provide data uniformly and we use it diversely?

GIS Reports Research

Desktop simulations and modelling

Cloud computing and many others...

The Spatial Information Services Stack Present day

Integration – WaterML 2.0

• Framework exists, we need a mechanism of sharing data between each other

• Under Australia’s 2007 Water Act, BoM collects observations of storage level and stream flow from over 200 providers across the nation.

• CSIRO has led the development of WaterML 2 standard specifically for quantity and is currently extending it for water quality as well.

• Time series data, allows near real time model-data integration

Geofabric – towards a single point of truth

Why?

• No consistent national scale water map of Australia

• Differences between States, regions, different resolutions

• Classic examples: • 5 different catchment boundaries for the same catchment

• Stream network does not match the DEM

• Stream network where the gauges are not on the streams

• Makes it difficult to compare and do basin level and national scale assessments

Hydrological Geofabric of Australia

• The Hydrological Geofabric provides: • A consistent spatial framework with

a historical gazetteer (location names);

• A specialised GIS that registers relationships between features from the hydrological system (rivers, lakes, reservoirs, dams, aquifers, drains and monitoring points)

• It also stores the agreed boundaries of basins, drainage divisions, catchments, aquifer and priority aquatic ecosystems.

User oriented & model driven products

Clouds – A watershed moment

Trends in technology

Cloud infrastructure

Software as a Service to support Research

• Provide modelling services through cloud computing

• Opportunistically applied to catchment modelling in Koshi

• eWater Source is the river system modelling software used

Source Modelling Service

Data Centre

Compute Node e.g. local machine

Source Modelling Service

Compute Node e.g. Azure

Compute Node e.g. Amazon

External Modeller uploading data and running models via the Web

CSIRO Scientist updating model science and functionality

With the Source Modelling Service, complex model runs and analysis can be undertaken from anywhere in the world and scaled to handle increasingly complex problems through use of commercial Cloud Providers

Remote Sensing Cloud

• Typically time consuming to find data and to process

• Very large data sets

• Require a subset of data often, (x,y,t)

• Earth Observations Data Cube • Spatially regular

• Calibrated images (cloud cover etc)

• Long time series (Landsat)

• Openly accessible using cloud technologies

Calibrated “Cubed” Data in AG-DC

Bringing it all together DEMO

Scenario – Water Sharing in the Ringarooma

Moorina Gauge: 6

cumecs e-flow Water Storage

Sensor Data Feeds

Managing flow

Sensor Cloud Conceptual Architecture

ACTION

KNOWLEDGE

INFORMATION

DATA

Sensor Cloud with real-time with eWater Source cloud modelling platform

DATA

Sensor Networks (real-time data)

Data Providers (spatial, historical)

APIs & Web Services

Apps (DSS)

Sensor Cloud - benefits

• Lots of projects under development in India: Hydrology projects – Sensor data, State Data, Central Data, Water Storage data, Climate data, River flow data, Ground water levels, Water Quality data & Historical data sets

• Don’t need expensive clusters in-house

• Easier to access and process parts of large data sets and much easier to share data and model outputs

• Bring them all together in to sensor cloud to support community engagement, government to citizen and government to government engagement

Next steps?

• Open data policy (real time and archived); Standards to support data exchange & standards based technology platforms (software and hardware)

• Capacity building to support river basin planning

• Build a community of practice, time, effort and focus. These are not easy things to implement. Focus efforts on an operational scenario.

Land & Water Flagship Amit Parashar

t +91 8130443332 E amit.parashar@csiro.au w www.csiro.au/

Thank you

LAND & WATER FLAGSHIP