copernicus & machine learning collaboration …...copernicus & machine learning:...

COPERNICUS & MACHINE LEARNING:Collaboration of ESA and the AI community

Sašo Džeroski

Jožef Stefan Institute, Ljubljana, Slovenia

Visiting professor, Φ-lab, ESA/ESRIN, Frascati, Italy

“Copernicus and Artificial Intelligence” workshop28 January 2020

Brussels, Belgium

Jožef Stefan Institute (JSI)Founded in 1949, JSI is the largest research institute in SloveniaPhysics, chemistry, molecular biology, biotechnology, information technologies, energy and environmentStrongest fields: Nano-, Eco- and IC- technologies

About 1000 employees, most with PhD degreeMany, many EU projects

European Research Ranking: In top 10 research institutes in Europe“Copernicus and Artificial Intelligence”

28 January 2020

Department of Knowledge TechnologiesFounded in 2004Approximately 40 researchers

Knowledge technologies = Information technologies that support the acquisition, management, modelling & use of knowledge and data

• Knowledge Management• Decision Support• Language Technologies• Knowledge Discovery: Machine Learning and Data Mining

“Copernicus and Artificial Intelligence”28 January 2020

My research groupSince 2005, I lead a research group @DKT/JSI, now cca. 20 people

Development of Machine Learning Methods for• Mining Big & Complex Data/Predictive Modeling• Computational Scientific Discovery• Ontologies for Data Mining & Comp. Sci. Discovery

Applications of Machine Learning• Life Sciences (incl. drug development, medicine)• Environmental Sciences (incl. agriculture, ecology, forestry)• Earth Observation (incl. agriculture, archeology)


ESA’s Φ-lab Founded in 2017, the Φ-lab (Phi Lab) at the European Space Agency has the mission of accelerating the future of Earth Observation

Helping Europe's • earth observation• space researchers, and • companies Adopt disruptive technologies and methods

Disruptive technologies & methods = Artificial Intelligence


ESA’s Φ-lab Some of the Φ-lab activities include • Building bridges between the EO and AI communities• Promotion of the use of Artificial Intelligence within ESA• Organizing the Φ-week, a highly successful/popular event• Organizing thematic workshops that bring the EO and AI

communities together (Φ-lab thematic workshops), such as• Atmospheric Water Cycling in a Changing Climate• AI, Machine Learning and Deep Learning and Archaeology

• Producing white papers, such as “Towards a European AI for Earth Observation Research & Innovation Agenda”


ESA’s Φ-lab People

Visiting professors program • Work with the team of Research Fellows (providing them with advice, ideas

and connections) • Helping ESA in shaping a Research & Innovation agenda for a AI4EO and a

strategy for ESA EO• Develop partnerships with a rapidly growing AI ecosystem.


The Artificial Intelligence Challenge of COPERNICUSCOPERNICUS is generating massive amounts of remotely sensed images in different modalitiesHigh quality, high resolution, dense temporal coverageThe data are publicly available

The challenge: Derive added value from these dataFacilitate access to these dataUse artificial intelligence techniques (ML) to infer additional information


Artificial Intelligence includes, but is not limited to Machine LearningAI is the science and engineering of making intelligent machines, especially intelligent computer programs • Knowledge representation• Knowledge engineering• Reasoning• Planning• Natural language processing• Computer vision


Deep LearningAbility to learn many-layered neural networks from vast amounts of data

Machine Learning includes, but is not limited to Deep Neural Networks• Reinforcement learning• Computational scientific discovery• Learning from data (inductive learning)

• Unsupervised learning (clustering)• Supervised learning (predictive modeling)

• Classification • Regression

• Learning decision trees & tree ensembles• Learning (if-then) rules & rule ensembles• Learning logic programs (relational rules)• Understandable/explainable models• Explainable AI


Neural Networks, Shallow and Deep


Neural Networks and Image Classification

Neural networks learn classification models, see patterns, spot anomalies. They can handle datasets far larger and messier than humans can cope with. They are great for classifying images!


Neural Networks and Satellite Image ClassificationNeural networks are also great for classifying satellite images!E.g., land cover classification from SENTINEL-2 imagesCf. BigEarthNet (group of Begum Demir)http://bigearth.net/

Sumbul et al., IGARSS, 2019


http://bigearth.net/

End-to-End-Learning: The key to the success of Deep NNs


The Limitations of DNNs: Data hunger

Deep Neural Networks need lots of labeled training data!BigEarthNet has 590326 Sentinel2 image patches

But COPERNICUS provides an abundance of data! So what’s the problem? This data is unlabeled!


The Limitations of DNNs: Computing power hungerAn example from a presentation at a recent scientific event Comparing the results of DNNs and Random Forests on a task of land cover classification

DNN performance: 97% accuracy, 3300 hrs GPURFs performance: 96% accuracy, 36 hrs CPU

Electricity spent (rough estimate): 1KWh for RFs, 1 MWh for DNNs


Transfer Learning with Deep Neural Nets


To combat scarcity of labeled data• learn in one domain (where enough

labeled data available), then transfer the learned knowledge to another

Fine-tuning pre-trained DNNs• Cut-off final layers of pre-trained net• Retrain final layers with labeled data

Why we shouldn’t use ImageNet pre-trained DNNs for classifying RS images?Slide by Begum Demir, Φ-week, September 2019


Pre-training NNs with unlabeled RS images

Deep Clustering: Unsupervised Learning of Visual Features

Rather than pre-training with labeled ImageNet, pre-train with unlabeled RS images/ patches


Pre-training NNs with unlabeled RS images:Preliminary ResultsPreliminary results on BigEarthNetland cover classification are great!


Machine Learning for explainable AI

Machine learning for big and complex data• Predicting structured outputs• On-line learning from data streams • Semi-supervised learningMaking machine learning an open science• Ontologies for describing machine learning (data types, ML tasks,

ML methods)• Ontologies for describing application domain


Multi-target prediction on data streams

• The dataset is not available in its entirety ahead of time• Theoretically infinite data examples

• Cannot store them: Processed once and discarded

• Often, real- or near real-time response is needed• Temporal dimension may lead to distribution change

• “Concept drift”


Descriptive space Target space

… … …

… … … … …

Example n 1 TRUE 0.49 0.69 0.68 0.60 3.91Example n+1 4 FALSE 0.08 0.07 0.10 1.69 7.57

Example n+2 6 FALSE 0.08 0.07 0.08 0.77 8.86Example n+3 8 TRUE 0.00 1.00 0.11 3.51 2.50Example n+4 6 TRUE 0.00 0.00 0.43 2.10 8.09

Example n+5 6 TRUE 0.46 0.11 0.56 0.99 7.59

Hierarchical Multi-label Classification


Semi-supervised Learning: Using Unlabeled Data


AI Prototyping Environment for Earth Observation• Joint effort with ESA• Slovenian actors: Bias Variance Labs, JSI, ZRC SAZU• Will develop tools for

• Unsupervised pre-training of DNNs with different types of RS data• Enabling effective extraction of features/ variables describing images• Making the pre-trained networks available like ImageNet pre-trained networks

• Will also develop tools allowing • The use of features/ variables defined by pre-trained networks • With other machine learning algorithms , e.g., tree-ensembles • For more complex tasks of predictive modeling, e.g., hierarchical classification


EO (incl. COPERNICUS) Data is BIG!But is it FAIR?

• Findable• Accessible• Interoperable• ReusableFor data to be FAIR, it needs to be accompanied with some knowledge (annotations) using terms from ontologies


Ontologies, Open Data Science, Reproducible AI/ML Research• Both data and SW need to follow FAIR principles

(Findable, Accessible, Interoperable and Reusable)• Ontologies for describing the elements and processes of data

science (data mining)• Data• Data Mining Tasks• Data Mining Algorithms• Data Analytics Processes/Workflows

• Together with domain ontologies, this allows• Precise description of the data analyses performed• Matching algorithms and data• Automated construction of data analysis workflows


Ontology of Data Mining/ Data Science


Directions for developmentDevelop AI tools and environments integrating different and powerful machine learning methods

Make COPERNICUS data accessible and AI-ready • Infrastructural/hardware aspects • Semantic/ annotation aspectsConnect remotely sensed with ground data• ‘Ground truth’ data are needed• Labeling/annotation efforts

High-performance Artificial Intelligence (as a Service)“Copernicus and Artificial Intelligence”

28 January 2020

Take-home Messages for the Successful use of AI / Machine Learning to COPERNICUS data1. INTEGRATE different approaches to machine learning

• Unsupervised/Supervised• Deep Learning/Tree ensembles

2. ANNOTATE satellite imagery (connect to ground data)• Need more AI-ready COPERNICUS data• For a variety of application domains

3. REUSE data, as well as models in a FAIR fashion• Use ontologies describing data, remote sensing, application areas to

annotate datasets• Share and re-use NNs pre-trained with unlabeled satellite images


Thanks!

To the ESA Φ-lab, and especially Sveinung Loekken, Pierre-Philippe Matthieu, and Iarla Kilbane-Dawe To my team and especially Dragi Kocev, Pance Panov, Nikola Simidjievski, and Ivica DimitrovskiThe complete Ljubljana eco-system, incl. JSI, JSI IPS, BV Labs


copernicus & machine learning collaboration …...copernicus & machine learning:...

Documents