Philipp Slusallek

German Research Center for Artificial Intelligence (DFKI)

Saarland University

Excellence Cluster Multimodal Computing and Interaction (MMCI)

European High-Level Expert Group on AI

Wie hilft Künstliche Intelligenz,

die Welt zu verstehen?

DFKI Research Area:

Agents and Simulated Reality (ASR)

Agents & Simulated Reality:AI & Graphics & HPC & Security

Research:

Topics &

Teams

Philipp Slusallek

Distributed & Web-

Based Systems

René Schubotz

Large-Scale Virtual

Environments

Philipp Slusallek

Multimodal

Computing

and

Interaction

High-Performance

Graphics & Computing

Richard Membarth

Distributed Realistic

Graphics

Philipp Slusallek

Computational

3D Imaging

Tim Dahmen

Knowledge- and Technology Transfer

VisCenter

Georg Demme

Strategic

Relations

Hilko Hoffmann

SW-Engineering &

Organization

Georg Demme

Scientific Director

Survivable Systems

and Services

Philipp Slusallek

Intelligent

Information Systems

Matthias Klusch

Multi-Agent

Systems

Klaus Fischer

Behavior, Interaction

& Visualization

Georg Demme

Visual

Computing

Philipp Slusallek

Application

DomainsAutonomous

Driving

Christian Müller

Industrie 4.0

Ingo Zinnikus

Smart

Environments

Hilko Hoffmann

Autonomous

Driving

Christian Müller

Smart System

Security

Andreas Nonnengart

High-Performance

Computing

Richard Membarth

Computational

Sciences

Tim Dahmen

Flexible Production Control

Using Multiagent Systems

Verification and Secure Systems

(BSI-certified Evaluation Center)Physically-Based Image Synthese

Scientific Visualisation GIS and Geo Visualization Reconstruction of Cultural Heritage

Future City Planning and Management Large 3D Models and Environments Large Visualization Systems

Intelligent Human Simulation in Production Web-based 3D Application (XML3D) Distributed Visualization on the Internet

ASR Research Topics

Multi-Agenten-Systeme:

Saarstahl, Völklingen

Flexible Production Control

Using Multiagent Systems

at Saarstahl, Völklingen

Multi-agent technology has been running a steelworks

24/7 for ~10 years now

Physically-Based Image Synthesis

with Real-Time Ray Tracing

Key product offered now by all major HW vendors:

Intel (Embree), Nvidia (OptiX), AMD, Imagination, …

DFKI Agenten und Simulierte Realität

7

Efficient Simulation of Illumination:

Light Propagation and Sensor Models

VCM now part of most commercial renders:

RenderMan, V-Ray, Corona, …

Large Visualization Systems Using Ray-Tracing

Several spin-off companies from our group:

inTrace, Motama, xaitment, Pxio, …

Real-Time Ray Tracing Processor [Siggraph’05]

Real-Time Ray Tracing Hardware is integral

part of every Nvidia GPU starting end of 2018

AnyDSL Compiler Framework

De

ve

lop

er

Computer

Vision

DSL

AnyDSL Compiler Framework (Thorin)

Physics

DSL…

Ray

Tracing

DSL

Various Backends (via LLVM)

Parallel

Runtime

DSL

AnyDSL Unified Program Representation

Layered DSLs

CPUs GPUs FPGAs

Verteilte Visualisierung im

Internet

• Mobile Visualization

• Jens Krüger

Display as a Service (DaaS Pxio):

Distributed Visualization on the Internet

Reconstruction of Cultural Heritage

Future City Planning and Management

DFKI Agenten und Simulierte Realität 14

Intelligent Human Simulation,

e.g. in Production Environments

Collaborative Robotics and

Simulated Reality

Autonomous Driving: Training

using Synthetic Sensor Data

Artificial Intelligence:

What is this?

What is AI?

• Artificial Intelligence (AI)

“Intelligence” exhibited by machines

Colloquially, the term "artificial intelligence" is applied when a machine mimics "cognitive functions” that humans associate with other human minds “

Study of “Intelligent/Autonomous Agents or Systems"

cf. Russell & Norvig (AI „Bible“), Wikipedia

What is AI?

• Elements of AI Systems: Perception

• … to find out what the world around “looks” like (incl. digital sources)

Machine Learning (Deep Learning)• … to automatically build/adapt models of reality

Knowledge Representation• … to represent, process, and communicate models of the environment

Reasoning• ... to infer and generalize new models based on existing knowledge

Planning• … to plan and simulate in the virtual world, exploring possible

options, evaluating their performance, and making decisions

Acting• … to manipulate and move around in the real/virtual world

Supporting• Validation/Verification, Security, Ethics/Social Aspects, SW/HW-Platform

Applications

DataScience

AI: Structure and Terminology

AI

Learning(Machine Learning)

Reasoning

Deep

LearningSVM …

Technologies

Applied AI Communities

Vision/

GraphicsLanguage Robotics

Rule-Based

Systems

Statistical

Reasoning…

Application Areas

Interaction Ethics …

Automotive Health Industrie4.0 Sciences Public Sector …

What is AI?

• AI Research: Three distinct points of view

AI as a Tool (what we have done successfully for 30+ years)• Enabling intelligent, supportive, and adaptive technology

• Systems aware of their environment and reacting accordingly

• AI should be perceived and handled as a tool!

• Humans decide and are responsible how these tools are used!

AI for Cognitive Sciences (we should do more of this)• Understanding the human brain and cognition

• Helps to better understand how we humans ”work”

AI for Artificial Humans (we should do NONE of this)• A non-goal for our and almost all other AI research

• Main cause of fear in the general public (sci-fi movies, etc.)

Understanding the World with AI

State of AI

• Amazing progress in recent years in all of AI Most visible due to Deep Neural Networks (DNNs)

Supported by more data, better HW, improved algorithms

• Success stories Perception: vision, speech, …

Game playing: Chess, Go, simple video games, …

Some complex tasks: translation, autonomous driving, …

• Challenges Where are the limits of AI? How to avoid the hype cycle?

Understanding and explanation of what is learned by AI

Modularity, integration with classical AI techniques

Validation and verification methods – building trust!

Why Do We Need Training and

Validation via Synthetic Data?

Example: Learning to Play Go

From Scratch via Simulations

• AlphaGo Zero [Nature, Oct´17]

Given: Rules + Deep-Learning + Reinforcement-Learning

Learning by simulation (here: playing against itself)

• But: In reality „rules“ are complex AND unknown!

Simultaneously learn rules, simulate, train & validate AIs

Quality of Gameplay

Autonomous Systems:

The Problem

• Our World is extremely complex Geometry, Appearance, Motion, Weather, Environment, …

• Systems must make accurate and reliable decisions Especially in Critical Situations

Increasingly making use of (deep) machine learning

• Learning of critical situations is essentially impossible Often little (good) data even for “normal” situations

Critical situations rarely happen in reality – per definition!

Extremely high-dimensional models

Goal: Scalable Learning from synthetic input data Continuous benchmarking & validation (“Virtual Crash-Test“)

Autonomous Driving:

The Problem

Learning from

real data

Typical situations

with high probability

Critical situations

with low probability

Learning from

synthetic data

Learning for

Long-Tail Distributions

Goal: Validation of correct behavior by

covering high variability of input data

Reality

• Training and Validation in Reality

E.g. driving millions of miles to gather data

Difficult, costly, and non-scalable

Reality

Car

Digital Reality

• Training and Validation in the Digital Reality

Arbitrarily scalable (given the right platform)

But: Where to get the models and the training data from?

Reality

Digital

Reality

CarCar

Teil-ModelleTeil-ModelleTeil-Modelle

Reality

Digital

Reality

Partial

Models

(Rules)

Modeling &

Learning

CarCar

Model L

earn

ing

Digital Reality: Learning

Geometry

Material

Behavior

Motion

Environment

…

SzenarienSzenarien

Teil-ModelleTeil-ModelleTeil-Modelle

Reality

Digital

Reality

Partial

Models

(Rules)

Relevant

Scenarios

Concrete

Instances of

Scenarios

Configuration &

Learning

Modeling &

Learning

Coverage of Variability via

Directed Search

CarCar

Model L

earn

ing

Reasoning

Digital Reality: Reasoning

SzenarienSzenarien

Teil-ModelleTeil-ModelleTeil-Modelle

Reality

Digital

Reality

Partial

Models

(Rules)

Simulation/

Rendering

Relevant

Scenarios

Concrete

Instances of

Scenarios

Configuration &

Learning

Modeling &

Learning

Synthetic Sensor Data,

Labels, …

Adaptation to the

Simulated Environment

(e.g. used sensors)

CarCar

Model L

earn

ing

Sim

ula

tion &

Learn

ing

Reasoning

Digital Reality: Simulation

Coverage of Variability via

Directed Search

SzenarienSzenarien

Teil-ModelleTeil-ModelleTeil-Modelle

Reality

Digital

Reality

Partial

Models

(Rules)

Simulation/

Rendering

Relevant

Scenarios

Concrete

Instances of

Scenarios

Configuration &

Learning

Modeling &

Learning

Synthetic Sensor Data,

Labels, …

Adaptation to the

Simulated Environment

(e.g. used sensors)

Car

Continuous

Validation &

Adaptation

Car

Model L

earn

ing

Reasoning

Validation / Adaptation / Certification

Digital Reality: Validation/Adaptation

Sim

ula

tion &

Learn

ing

Coverage of Variability via

Directed Search

Coverage of Variability via

Directed Search

SzenarienSzenarien

Teil-ModelleTeil-ModelleTeil-Modelle

Reality

Digital

Reality

Partial

Models

(Rules)

Simulation/

Rendering

Relevant

Scenarios

Concrete

Instances of

Scenarios

Configuration &

Learning

Modeling &

Learning

Synthetic Sensor Data,

Labels, …

Adaptation to the

Simulated Environment

(e.g. used sensors)

Car

Continuous

Validation &

Adaptation

Car

Model L

earn

ing

Reasoning

Validation / Adaptation / Certification

Digital Reality: Continuous Learning

Continuous Learning Loop for AI

Not just for Automated Driving:

Works for any AI System

where we can model its interaction with

the environment

Sim

ula

tion &

Learn

ing

Digital Reality for

Autonomous Driving (DFKI)

Intelligent Simulated Reality for

Autonomous Systems (BMW)

Using PreScan Data (TASS)

for Semantic Segmentation

Challenges: Better Models of

the World (e.g. Pedestrians)

• Long history in motion research (>40 years)

E.g. Gunnar Johansson's Point Light Walkers (1974)

Significant interdisciplinary research (e.g. psychology)

• Humans can easily discriminate different styles

E.g. gender, age, weight, mood, ...

Based on minimal information

• Can we teach machines the same?

Detect if pedestrian will cross the street

Parameterized motion model & style transfer

Predictive models & physical limits

Challenge: Better Simulation

(e.g. Radar Rendering)

• Key Differences

Longer wavelength: Geometric optics (rays) not sufficient

Need for some wave optics

• Diffraction at rough surfaces and edges

• Need for polarization & resonance

Highly different goals

• Optical: Focus on diffuse effects (+ some highlights, reflections, etc.)

• Radar: Focus on specular transport only (i.e. caustic paths)

• Recent Work on Caustics [Grittmann et al., EGSR’18]

Identifying “useful” specular paths (using VCM)

Guides samples to visible specular effects (e.g. indirect radar echos)

• Combining research on rendering and radar technology

CLAIRE

CONFEDERATION OF LABORATORIES FOR ARTIFICIAL INTELLIGENCE RESEARCH IN EUROPE

Excellence across all of AI. For all of Europe.

With a Human-Centred Focus.

claire-ai.org

Take-Aways

• Digital Reality as a fundamental tool in AI Modeling and simulation even in complex environments Learning and reasoning via feedback loop (e.g. RL) Key element for training & validation of future AI systems

• Continuous Learning Loop using Synthetic and Real Data Loop of model learning, simulation, training, and validation Validation is required to establish trust in AI systems

• Big Challenges Ahead Many promising partial results already – but largely islands Requires closer collaboration of industry & academia CLAIRE: Towards large-scale European initiative

AI: A Central Component of our Future for Many Years