22 - csiro - water data management-sep-17
DESCRIPTION
22 - CSIRO - Water Data Management-Sep-17TRANSCRIPT
Water data management platforms Modern Tools & Techniques for Water Resources Assessments & Management
Amit Parashar
17 September 2014 New Delhi
LAND & WATER FLAGSHIP
Contents
• CSIRO
• Data management challenges
• Data sharing platforms
• Hydrological geofabric
• Cloud computing
• Demo
Top 1% of global research
institutions in 14 of 22 research
fields Top 0.1% in 4 research fields
Darwin
Alice Springs
Geraldton 2 sites
Atherton
Townsville 2 sites
Rockhampton
Toowoomba
Gatton
Myall Vale Narrabri
Mopra
Parkes
Griffith
Belmont
Geelong
Hobart Sandy Bay
Wodonga
Newcastle
Armidale 2 sites
Perth 3 sites
Adelaide 2 sites Sydney 5 sites
Canberra 7 sites
Murchison
Cairns
Irymple
Melbourne 5 sites
CSIRO: Who we are
Werribee 2 sites
Brisbane 6 sites
Bribie Island
People
Locations
Flagships
Budget
6000
58
9
$1B+
Challenges in data management
• Lots of investment in water data collection (collecting, finding, accessing and formatting of data)
• Multiple agencies collecting data – different ways of managing data
• No single point of truth (overlaps between agencies)
• Fragmented data silos
• Difficult to aggregate quality assured data set as input to research and inform policy
Challenges in data management
• Supporting information/data requests again and again and again (usually for the same data) Need scaleable mechanism to share data
• Water management pressures require increasing complex and integrated assessments to support decision makers.
• As India moves towards IWRM & Basin Level Planning, water data managers will increasingly need to support a variety of jurisdictions, agriculture, urban, environment, energy
• Community pressure for access to data (increased transparency)
Discover Access Understand Extract, Transform, Load
Use
Time and effort
Research Focus To enable more efficient and effective management of water by improving the availability, accessibility and usability of existing and new water information products and services.
Action
Knowledge
Information
Data
Action
Knowledge
Information
Data
Spatial Information Service Stack – Data Sharing Platforms
Discovery/Access/Integrate
Why? Provide data uniformly and we use it diversely?
GIS Reports Research
Desktop simulations and modelling
Cloud computing and many others...
The Spatial Information Services Stack Present day
Integration – WaterML 2.0
• Framework exists, we need a mechanism of sharing data between each other
• Under Australia’s 2007 Water Act, BoM collects observations of storage level and stream flow from over 200 providers across the nation.
• CSIRO has led the development of WaterML 2 standard specifically for quantity and is currently extending it for water quality as well.
• Time series data, allows near real time model-data integration
Geofabric – towards a single point of truth
Why?
• No consistent national scale water map of Australia
• Differences between States, regions, different resolutions
• Classic examples: • 5 different catchment boundaries for the same catchment
• Stream network does not match the DEM
• Stream network where the gauges are not on the streams
• Makes it difficult to compare and do basin level and national scale assessments
Hydrological Geofabric of Australia
• The Hydrological Geofabric provides: • A consistent spatial framework with
a historical gazetteer (location names);
• A specialised GIS that registers relationships between features from the hydrological system (rivers, lakes, reservoirs, dams, aquifers, drains and monitoring points)
• It also stores the agreed boundaries of basins, drainage divisions, catchments, aquifer and priority aquatic ecosystems.
User oriented & model driven products
Clouds – A watershed moment
Trends in technology
Cloud infrastructure
Software as a Service to support Research
• Provide modelling services through cloud computing
• Opportunistically applied to catchment modelling in Koshi
• eWater Source is the river system modelling software used
Source Modelling Service
Data Centre
Compute Node e.g. local machine
Source Modelling Service
Compute Node e.g. Azure
Compute Node e.g. Amazon
External Modeller uploading data and running models via the Web
CSIRO Scientist updating model science and functionality
With the Source Modelling Service, complex model runs and analysis can be undertaken from anywhere in the world and scaled to handle increasingly complex problems through use of commercial Cloud Providers
Remote Sensing Cloud
• Typically time consuming to find data and to process
• Very large data sets
• Require a subset of data often, (x,y,t)
• Earth Observations Data Cube • Spatially regular
• Calibrated images (cloud cover etc)
• Long time series (Landsat)
• Openly accessible using cloud technologies
Calibrated “Cubed” Data in AG-DC
Bringing it all together DEMO
Scenario – Water Sharing in the Ringarooma
Moorina Gauge: 6
cumecs e-flow Water Storage
Sensor Data Feeds
Managing flow
Sensor Cloud Conceptual Architecture
ACTION
KNOWLEDGE
INFORMATION
DATA
Sensor Cloud with real-time with eWater Source cloud modelling platform
DATA
Sensor Networks (real-time data)
Data Providers (spatial, historical)
APIs & Web Services
Apps (DSS)
Sensor Cloud - benefits
• Lots of projects under development in India: Hydrology projects – Sensor data, State Data, Central Data, Water Storage data, Climate data, River flow data, Ground water levels, Water Quality data & Historical data sets
• Don’t need expensive clusters in-house
• Easier to access and process parts of large data sets and much easier to share data and model outputs
• Bring them all together in to sensor cloud to support community engagement, government to citizen and government to government engagement
Next steps?
• Open data policy (real time and archived); Standards to support data exchange & standards based technology platforms (software and hardware)
• Capacity building to support river basin planning
• Build a community of practice, time, effort and focus. These are not easy things to implement. Focus efforts on an operational scenario.
Land & Water Flagship Amit Parashar
t +91 8130443332 E [email protected] w www.csiro.au/
Thank you
LAND & WATER FLAGSHIP