ShareCam: Interface, System Architecture, and Implementation of a

Collaboratively Controlled Robotic Webcam

Dezhen Song (TAMU) and Ken Goldberg (UC Berkeley)

IEEE/RSJ International Conference on Intelligent Robots and SystemsFinalist for New Technology Foundation (NTF) Award for

Entertainment Robots and Systems

Teleoperation: Related Work• Tesla, 1898• Goertz, ‘54• Mosher, ‘64• Tomovic, ‘69• Salisbury,Bejczy, ‘85• Ballard, ’86• Volz, ’87-• Sheridan, ‘92• Sato, ’94• Goldberg, ’94-• Presence Journal ‘92-• O. Khatib, et al. ’96

Internet

networkedrobot:

Taxonomy (Tanie, Matsuhira, Chong 00)

Multiple Operator, Single Robot (MOSR):

Single Operator, Single Robot (SOSR):

Single Operator, Multiple Robot (MOSR):

Pan, Tilt, Zoom robotic video camera

sharecam

Entertainment Applications

Playing Games

Related Work• Networked robots

– Tanie, K., Chong, N. et al(01)– Jia, S. and K. Takase (01)– Hu, H., Yu, L., Tsui, P., Zhou, Q (01)– Safaric, R. et al. (01)– Goldberg and Siegwart (02)– Coppin, P. and Wagner, M.D. (02)– Konukseven, I., Erkmen, A. et al (02)

• SOSR– Siegwart, R. and Saucy P. (99)– Paulos, E. and Canny, J. (99)– Tanie, K., Arai, H. et al. (00)– Lynch, K. and Liu, C. (00)– Fong, T., Thorpe, C., et al(01)

Related Work• SOMR

– Hu, Yu, Tsui, Zhou (01) – Jia, Takase (01)

• MOMR– Fukuda, Xi, Liu, Elhajj et al. (00,02)– Tanie, Chong, et al. (00)

• MOSR– Cinematrix (91)– Cannon, McDonald, et al. (97) – Goldberg, Chen, et al. (00, 01)

Sharecam: System Architecture

Users

Internet

ShareCam Server

Video Server

Sharecam Software

TCP/IP

TCP/IP

User database

RegistrationCore (with shared memory segments)

Apache module

Apache module

Apache module

Communication

Console/Log Login CGI

ShareCam web server

ShareCam applet InetCam applet

Client

RS232C

HTTP

Camera control

Calibration

Panoramic image generation

InetCam server

Video server

Canon VC-C3 Camera

Java

Gnu C++

PERL

MySQL

frame selection problem: given n requests, find optimal frame

Problem Definition

Requested frames: i=[xi, yi, zi], i=1,…,n

Problem Definition• “Satisfaction” for frame i: 0 Si

1

Si = 0 Si = 1

= i = i

•Symmetric Difference

•Intersection-Over-Union

SDArea

AreaIOU

i

i

1)(

)(

)(

)()(

i

ii

Area

AreaAreaSD

Similarity Metrics

Nonlinear functions of (x,y)

Intersection over Maximum:

),(

)(

),max(

)1,)/min(()/(),(

i

i

i

i

biiii

Max

Area

aa

p

zzaps

Requested frame i , Area= ai

Candidate frame

Area = api

• global satisfaction:

n

iii

n

i

biii

yxpyxS

zzapS

1

1

),(),(

)1,)/min(()/()(

for fixed z

find * = arg max S()

approximation

n)dd

whgO(

spacing resolution :

spacing lattice :

z2

zd

d x

y

d

Compute S(x,y) at lattice of sample points:

w, h : width and height,g: size range

error bound

run time:O(n / 3 )

c* Optimal frame

Optimal at latticec~

Smallest frame on lattice that encloses c*

c

)ˆ()~()( * cscscs

)(

)ˆ(

)(

)~(1

** cs

cs

cs

cs

zdz

z

2...

min

min

1

Processing Zoom Type Complexity

Centralized Discrete Exact O(n2)

Centralized Discrete Approx O(nk log(nk)), k=(log(1/ε)/ε)2

Centralized Contin. Exact O(n3)

Centralized Contin. Approx O((n + 1/3) log2 n)

Distributed Discrete Exact O(n), Client: O(n)

Distributed Contin. Approx O(n), Client O(1/3)

frame selection algorithms

ShareCam Application:Game based learning : global environment

robotic video cameras

motion sensors

timed checks

sensor networks

humans: amateurs and profs.

Collaborative Observatories for Natural Environments (CONEs)

Frank van der Stappen (CS, Utrecht)Vladlen Koltun (EECS, UC Berkeley)George Bekey (CS, USC)Karl Bohringer (CS, UW)Anatoly Pashkevich (Informatics, Belarus) Judith Donath (Media Lab, MIT) Eric Paulos (Intel Research Lab, Berkeley)Dana Plautz (Intel Research Lab, Oregon)Sariel Har-Peled (CS, UIUC)

Thank you.ShareCam: Interface, System Architecture, and Implementation of a

Collaboratively Controlled Robotic Webcam

Satellite Imaging

MIT Press, 2002

Networked Robots

• Tele-Operation• Internet Tele-Operation• Collaborative Tele-Operation• Tele-Actor• Co-Opticon• Co-Opticon Algorithms

www.ken.goldberg.net

Infiltrate

NetworkedRobots

internettele-robot:

RoboMotes: Gaurav S. Sukhatme, USC

Smart Dust: Kris Pister, UCB (Image: Kenn Brown)

NetworkedCameras

Where to look?

Sensornet detects activity• “Motecams”

• Other sensors:audio, pressure switches,

light beams, IR, etc

• Generate bounding boxes

and motion vectors

• Transmit to PZT camera

Activity localization

1. Network Standards: HTML, Browsers, Java

2. Infrastructure: Backbone, Routers, ADSL

3. Public Adoption

4. Bandwidth: 10 Mbps, 100Mbps, Gbps

5. Video/Audio Compression: MP2,3,4

Networked robots Systems that couple communication to control of one or more robots in a network that often includes sensors and remote human operators.

Two subclasses : 1) Tele-operated, where human supervisors send commands

and receive feedback via the network. Such systems support research, education, and public awareness by making valuable resources accessible to broad audiences.

2) Autonomous, where robots and sensors exchange data via the network. In such systems, the network extends the effective sensing range of individual robots.

Challenges:

1. Variable Time Delays, Congestion

2. Latency

3. Access Control, Security, Interfaces

4. Protocol Design

5. Noise, Error Detection and Recovery

6. Deployment, Dynamic Routing

7. Power Management

8. Hybrid Architectures

Conventional Security Cameras

• Immobile or Repetitive Sweep• Low resolution

Future Work

• Continuous zoom (m=)• Multiple outputs:

– p cameras – p views from one camera

• “Temporal” version: fairness– Integrate si over time: minimize accumulated

dissatisfaction for any user

• Network / Client Variability: load balancing

• Obstacle Avoidance

Outline

• Collaborative Teleoperation

• Cinematrix• Co-opticon• Tele-Twister

"In times of terror, when everyone is something of a conspirator,everyone will be in a situation where he has to play detective." -- Walter Benjamin (1938)

Statistics of Satellite Imaging

• 2.5 Billion Market in 2003• Increasing 300% per year

since 1999• Major clients

– Government / Military– Oil exploration– Weather Prediction– Agriculture

Ikonos, 1999

Intersection over Maximum: si( ,i)

si = 0.20 0.21 0.53

Requested frame i

Candidate frame

• Staircase Approximation– Exact algorithm O(n3/2 log3 n) [data

structure]

– Approximation Algorithm O(nk log(nk)), k=(log(1/ε)/ε)2

• Staircase approximation, large constant factors