The Retinal Sub-surface Imaging Testbed @ Rensselaer:

3-D Spatial Mapping & Referencing Problems Faculty: Prof. Badri Roysam (RPI), Prof. Charles V. Stewart (RPI),

Collaborators: Dr. James N. Turner (RPI/Wadsworth), Howard L. Tanenbaum, MD, (Center for Sight) Graduate Students: Ali Can (RPI), Hong Shen (RPI), Kripa Rajashekhar (RPI)

Undergraduates: Ameesh Makadia (RPI), Jesse Raymond (RPI), DJ Wilsey (RPI)

Goals & Barriers The goal is to develop spatial mapping and referencing technologies for diverse sub-surface imaging problems. Much like Global Positioning Satellites (GPS) have enabled novel commercial and military capabilities, these methods will enable a generation of highly capable “spatially aware” instruments for sub-surface operations. E.g., in the medical context, this technology can be used to guide surgical tools, monitor treatment dosages, detect and track changes to tissue, provide safety shutoffs and alarms, and construct virtual environments for surgical planning and training. To date there are no successfully accepted surgical systems based on real-time computer vision. To break this barrier, vision systems that can operate accurately, fault tolerantly, and predictably over extended durations in the context of high scene complexity, varying image quality, and modeling limitations are needed. Notwithstanding these complexities, the systems must be totally ‘transparent’ and nearly invisible to the users.

Significance There is a compelling need to reduce the failure rate (≈ 50%) in laser retinal surgery. This is the best-available treatment for the leading causes of blindness affecting 25-30 million in U.S. alone. Broader applications within ophthalmology include automatic functional mapping of the retina for glaucoma, change detection, and automated scoring of clinical trial images. Most sub-surface images must eventually be related spatially with surface images for action planning.

Technical Approach • Integrated instrumentation for diagnosis and

surgery: A multi-spectral imaging system is that can capture images of multiple layers of the retina. ICG images from infrared wavelength is used for diagnosis, while red-free images from visible-wavelength is used for spatial referencing in real-time surgery.

• Progressive, Exploratory feature extraction: Extracts a sequence of partial results that contain high quality features needed for registration and referencing without visiting all the pixels.

• Spatial mapping of the curved retina: Robust hierarchical vision algorithms to mosaic the curved retina from projections.

• Real-time spatial referencing: Fast indexing algorithms to identify feature correspondences. Based on this, the spatial transformation between the live images that are captured during surgery and the wide-area retinal map are computed constantly and in real-time.

Objective Lens

ICG Barrier Filter (pass > 805nm)

Dioptic Correction Lens Focusing

Lens

Picture Angle Lens

QTH Lamp

Visible-Spectrum

CCD

Ocular Lens

Integrating Sphere

795nm Excitation Laser Diode

Laser Line Filter (795nm, BW 10nm)

Red-Free Filter (510-600nm)

��������������

��������������

����������������

������������������������������������������������

Near-Infrared CCD

����������

Dichroic Filter

ON/OFF mirror

Beam Mixer

Real-Time Image Processor

Collimator Lens

Display and Interface

�������� 795nm

Surgical Laser(Diode)

Y-axis Steered, Dot-Silvered Glass

Joystick

Servo

> 650nm

< 650nm

Center Stop

��������������

1-to-1 Relay

Fiber Optic

X-axis Steered Mirror

Model Eye

x-y stage controlled by a separate PC.

3-D Rotation Apparatus for eye model Electric shutter to simulate blinks

Tilted mirror with imaging aperture

Figure 1: (Top) Retinal testbed. (Middle) Current setup. (Bottom) Retinal surface image; Sub-surface image; & Overlay of the surface image onto the sub-surface image.

Relation to NSF ERC Related spatial mapping and referencing problems occur in all sub-surface imaging problems. They are particularly relevant when it is desired to image a much larger region of space than can be acquired by the sensor, and whenever it is desired to plan and execute specific actions (e.g., tool guidance, navigation, surgery.)

Current Status Algorithms for mosaicing the curved retina from projections, and 3-D confocal images are now well developed within this group. Effective 2-D referencing algorithms have been developed. We’re currently working on full 3-D reconstruction & indexing based rapid 3-D spatial referencing algorithms. Parts of the instrumentation have been built.

Plans and Project Evolution The retinal testbed will be assembled during the first year. The methods will be generalized progressively over the next three years in the context of other applications within the ERC. The model eye will be replaced with other models. A website will disseminate spatial mapping and referencing code. A fully working clinical trials-ready prototype will be demonstrated in 3 years. Longer term, we plan to make spatial mapping and referencing a core capability in a variety of intelligent “spatially aware” instruments in ophthalmology and beyond.

Key References [1] "Rapid automated tracing and feature extraction from live high-resolution retinal fundus images using direct exploratory algorithms," IEEE Trans. on IT in Biomed., vol. 3, no. 2, pp. 125-138, June 1999 [2] “Image Processing Algorithms for Retinal Montage Synthesis, Mapping, and Real-Time Location Determination," IEEE Trans. BME, vol. 45, no. 1, pp. 105-118, January 1998. (Reprinted in IMIA Yearbook, 1999).

[3] "Robust Hierarchical Algorithm for Constructing a Mosaic from Images of the Curved Human Retina," Proceedings IEEE –CVPR Conf., vol. 2, pp. 286-292, Fort Collins, Colorado, June 1999.(Best Paper Award) [4] “Optimal Scheduling of Tracing Computations for Real-time Vascular Landmark Extraction from Retinal Fundus Images,” submitted to IEEE Trans. on IT for Biomedicine. [5] "A feature-Based Technique for Joint, Linear Estimation of High-Order Image-to-Mosaic Transformations: Application to Mosaicing the Curved Human Retina," submitted to IEEE-CVPR conf., June 2000.

PI Contact Information Badrinath Roysam, Associate Professor, ECSE and Biomedical Engineering Depts., Rensselaer Polytechnic Institute, Rm JEC 6046, 110, 8th Street, Troy, NY 12180; Phone: 518-276-8067; Fax: 518-276-6261; Email: roysam@ecse.rpi.edu

Other Connections The Center for Sight provides data, and medical guidance. The Wadsworth Center is involved in instrument development, especially testing. The Woods Hole Oceanographic Institute will collaborate on oceanographic applications. The Scheie Eye Institute (Philadelphia) will drive other ophthalmic applications, & clinical trials.

x

y z

x '

y ' z '

u '

v '

u

v P

p

p '

Retina

Lens

Iris

Cornea

Reference Camera Coordinate System

Optic Disk

Choroid

Vitreous Humour

Fig. 2a: Imaging geometry.

Fig. 2b: Layered retinal structure.

Fig. 2c: Retinal Landmarks.

Fig. 3: (Left) Image frame. (Right) Retinal mosaic.