cvvr lab.@ national dong hwa univ. 國立東華大學 vr output

CVVR Lab.@ National Dong Hwa Univ. 國立東華大學

VR OutputVR Output


VR Output

人類 (Human) 電腦 (Computer)

1. 視覺 (See) 視訊 (Video,Image), 文字(Text)

2. 聽覺 (Hear) 音訊 (Audio)3. 觸覺 (Touch) 力回饋 (Force feedback)4. 味覺 (Taste)5. 嗅覺 (Smell)


Human eye

角膜虹膜

水晶體睫狀肌

視網膜

鞏膜

脈胳膜

玻璃液

神經節細胞 (100萬個 )

中央凹

瞳孔

Understanding the human

visual system is

necessary to design or

select a graphics

display


Cones & Rods圓錐細胞 & 圓柱細胞

( day & night)( 夜盲 vs. 色盲 )


The Human Visual System

• More than 126 million photoreceptors that are unevenly distributed over the retina

• Fovea• The central area of the retina• Several degrees around the eye’s viewing axis• High resolution and color perception• Focus area

• Areas that surround the fovea• Low-resolution• Motion perception photoreceptors

• We do not know what portion of the display is viewed by the fovea– The whole scene needs to be rendered at high resolution– A waste of graphics processing– The eye tracking technology is at present too bulky to be integrated with personal

displays


Field of view

• One eye• Approximately 150o horizontally and 120o vertically

• Both eyes• Approximately 180o horizontally and 120o vertically

• Central portion of the viewing volume• The area of stereopsis

• Both eyes register the same image

• Approximately 120o horizontally

• The brain measures depth


Each eye sees a

different field of

view


Stereoscopic vision

• The convergence angle• The angle between the viewing axis and the line to the

fixation point

• The angle depends also on the interpupillary distance (IPD)

• Interpupillary distance (IPD)• The distance between the pupils of the right and left eyes

• Varies among male and female adults within a range of 53-73 mm

• The baseline from which a person interprets distances to

objects

• The larger the IPD, the larger is the convergence angle


Stereoscopic vision

• Image parallax• The fixation point appears shifted horizontally between the

right and left eyes• The different position in relation to the two eyes• Must be replicated to help the brain interpret depth in the

simulated world created by VR graphics and stereo viewing hardware

• Stereoscopic displays• Need to output two slightly shifted images• Two displays (e.g., HMD): present its image to thecorresponding eye• Single display: time-sequenced or spatially sequenced


Depth perception

• Stereopsis is depends on distances to objects• Good for short distances, where the image parallaxis substantial• As objects get further away from the viewer, theirhorizontal shift gets smaller• Accuracy is degraded substantially at 10m from theuser

• Based on cue inherent in the image• Linear perspective, shadows, occlusions, surfacetexture, object detail, and motion parallax


Personal Graphics Displays

• A graphics display that outputs a virtual scene destined to be viewed by a single user

• Types: monoscopic or stereoscopic

• Category– Head-mounted displays (HMD)

– Hand-supported displays (HSD)

– Floor-supported displays

– Autostereoscopic monitors


Head-Mounted Displays (HMD)

• Project an image floating some 1-5 m in front of the user• The lower the HMD resolution and the higher the FOV The

greater is the number of arc-minutes of eye view corresponding to each pixel

• Consumer-grade HMDs– Usually LCD displays– TV programs and video games: NTSC/PAL monoscopic video input– VR system: Require the conversion of the graphics output signal RGB

format to NTSC/PAL

• Professional-grade HMDs– Usually CRT displays– Specifically for VR: Accept RGB video input


• Can display either stereo or mono images

• Stereo images (binocular disparity) • Same image twice (binocular concordance) • Single image (uniocular)

• May be totally immersive or semi-immersive (see-through) • May include a built-in head-tracker • May have built-in stereo headphones

Head Mounted Displays


VPL Eyephone & Olympus Eye-trek

• The VPL EyePhone in early 1990s– The first LCD-based HMD

– Resolution of 360 x 240

– FOV of 100o horizontally and 60o vertically

– 2.4 kg

– Induce user fatigue

• The Olympus EyeTrek in 2000– Resolution of 267 x 225 (FMD200)

– 30o horizontally and 22.7o vertically

– 100 grams


Dayang Cy-visor DH-4400VP

• Liquid-crystal-on-silicon (LCOS) display• Higher electrode densities (e.g., display resolution)

• Reduced power consumption

• Reduced volume

• Reduced manufacturing costs

• SVGA-resolution (800 x 600)• 60o horizontal x 43o vertical FOV• 160 grams


ProView XL35

• The professional grade LCD-based HMD• High-resolution AMLCD displays• The support allows for the placement of a tracker close

to the top of the head• 28o horizontal x 21o vertical stereo: 100% overlap• XGA resolution (1024 x 768 x 3 pixels)• Very low virtual image granularity: 1.6 arc-minutes/color

pixel• Adjustable IPD (55-75 mm)• Allow eyeglasses to be worn together• 992 grams, $19,500


Datavisor HiRes

• Two monochrome miniature CRTs• 1280 x 1024 resolutions• 78o horizontal x 39o vertical FOV• IPD (58-73 mm)• Focal distance (from 0.25 m to infinity)• 1.9 arc-minutes/color pixel• 1587 grans, $35,000


Virtual Retinal Display (VRD)

• by Microvision • Image is projected directly onto the retina


• Video see-through AR– the computer graphics are

overlaid on video of the real world

– TriVisio ARvision-3D

• Optical see-through AR– the virtual images are overlaid

directly on the real world – MicroOptical HMV-V2


HMD of the future?

•Built by Toshiba in 2006•120° horizontally •70° vertically •40 centimetre dome-shaped screen•3 Kg

•Looking more like the helmet from Armstrong's space suit than the must-have product of tomorrow


Hand-Supported Displays (HSD)

• Personal graphics displays that the user holds in one or both hands in order to periodically view a synthetic scene

• The user can go in and out of the simulation as required by the application

• Hardware• – Similar to HMDs• – Push buttons used to interact with the virtual

scene


Virtual Binocular SX• Virtual Binocular SX by NVIS Inc.• Constructed to resemble the look and feel of regular binoculars• Allows variable distance focusing as well as zooming on the scene• Two miniature LCOS displays• A tracker to measure the user’s viewing direction• High-resolution image (1280 x 1024)• Low granularity (1.6 arc-minutes/pixel)• Large weight (about 1 kg, without the tracker)• High unit cost ($19,900)


Floor-Supported Displays

• HMDs and HSDs could cause simulation sickness from large time delay between corresponding head and image motions

• The use of mechanical tracker: Almost instantaneous response to the user’s head motion

• Characteristics of floor-supported displays– An articulated mechanical arm to offload the weight of the

graphics display from the user– Integrate sensors directly in the mechanical support

structure holding the display– Offer larger FOV and superior graphics resolution than

HMDs or HSDs


• By Fakespace• Uses a CRT to provide high-resolution display • It is comfortable to use, since it does not have to be worn • Has fast, accurate, built-in tracking

BOOM (Binocular Omni-Orientation

Monitor)


BOOM3C• Fakespace Labs• A position resolution of 0.1o

• The latency is negligible (0.2 msec)• Allows intermittent use of computer

keyboards• Less freedom of motion due to a dead zone

created by the supporting pole• Work envelope: A hollow cylinder of 1.8 m

diameter and 0.8 m height• Graphics resolution: 1280 x 1024• Considerably larger FOV than that of the

Datavisor HMD


Window VR

• By Virtual Research Systems• A high-resolution LCD flat panel display: 21 in. diagonal, 1600 x 1200 pixels• Rely on third-party 3D trackers (InterSense 300 inertial trackers) that inherits

the problems associated with latencies and tracker noise• Only monoscopic graphics can be displayed• Two handles with pushbuttons and switches• The weight is supported by cables connected to an overhead

counterbalanced arm• The simplicity of the device and the lack of complex optics make it intuitive

to use: The feeling is that of viewing the simulation through a window• No restrictive space inside the work envelope: The supporting arm is off-axis


Desk-Supported Displays

• HMDs and HSDs– Excessive display weight causes fatigue to users (e.g.,

neck and arm pain)

• Floor-supported displays– Excessive weight increases inertia when the display is

rotated– Can cause unwanted pendulum oscillations

• Desk-supported displays– Fixed and designed to be viewed while the user is sitting:

Weight is not an issue– The user’s freedom of motion is limited when compared to

HMDs or HSDs


Autostereoscopic displays

• Produce a stereo image while viewed with unaided eyes

• Do not wear any vision apparatus• “column-interlaced” image format• Alternates individual columns assigned to the left-eye

view and the right-eye view• Reduced horizontal image resolution• Increased system complexity and increased cost• Two types: Passive & Active


• Do not track the user’s head• Restrict the user’s eyes to be within a small area for the stereo perception to be realized• The relation between the backlighting distance d and the distance to the user D determines a stereo viewing cone• DTI 2018XL Virtual Window & Sharp LL-151D

Passive autostereoscopic display


Active autostereoscopic display

• Tracking the user’s head and performing a real-time adaptation of the column interlaced image• Alleviate the problem of limited viewing volume in passive autostereoscopic display• Allows changes in viewing angle as large as ±25o to be accommodated when the user is 65 cm from the display• Apply mechanical means to adapt the column-interlaced image to the user’s head position• Ecomo4D display by Elsa


Major drawbacks of personal graphics displays

• Hard to justify the high price of professional grade products for one viewer

• There are cases when more than one user has to look at the same image– Share ideas– Work collaboratively


Large-Volume Displays

• Graphics displays that allow several users located in close proximity to simultaneously view an image of the virtual world

• Improves users’ freedom of motion and natural• interaction capability compared to personal

displays• Classifications

– Monitor-based (single or side-by-side CRTs)– Projector-based: Workbenches, CAVEs, display walls,

and domes


Monitor-Based Large-Volume Displays

• Use the stereo-ready monitor• Capable of refreshing the screen at double the

normal screen rate: 120-140 scans/sec• Each user wears a set of shutter glasses

– An infrared (IR) emitter located on top of the CRT display controls the active glasses

– Close and occlude one eye or the other alternately

• A head tracker can be added to the system– Change the image according to the user’s viewing direction– Placed on top of the monitor


Characteristics of Monitor-Based Large-Volume Displays

• The image is much shaper than that of LCD-based HMDs

• Not tiring even for long simulation times• The image is less luminous than on a normal screen• Require the direct line of sight between the IR emitter

and the IR received• Only one person can control over the virtual scene• Work as a window to the virtual world• Limited working envelope• The user’s FOV grows with the reduction in the

distance to the monitor and with the size of the screen


• Display shows left and right images alternately, switching at high speed between images • CRT monitor is required with double the normal scan rate• Infrared (IR) emitter can be used to synchronize the active glasses in a wireless mode • Typical 'Fishtank VR' • Particularly good for large audiences in a theatre

Liquid Crystal Shutter (LCS)


Panoram

• PV290 display by Panoram Technologies• Three TFT panels, each with a resolution of 1280 x 1024 pixels• Compound image with 3840 x 1024 pixels: Require proper synchronization


Projector-Based Displays

• Allow many closely located users to participate in a VR simulation

• CRT projectors to produce the stereo-pair image– More recently digital projectors replaced CRT projectors

• Frame sequential mode– The projector splits the number of scan lines in two– The user wearing active glasses sees a stereo image

• Drawbacks– Inability to project bright images– Cost


• Workbench by Fakespace in 1998

• The first projector-based large-volume displays

• Project 3D scene on a large horizontal table-size display

• V-Desk 6 by Trimension in 2001

Workbench & V-Desk


Stereo viewing cone of Workbench

• Several viewers wearing active glasses can simultaneously see 3D objects floating on top of the workbench in a stereo viewing cone.– This cone depends on the position of the user and the

degree of tilting of the workbench surface.– If the workbench is horizontal, tall 3D objects will be clipped

at the side opposite the user

• The stereo collapsed effect• Modern workbenches allow the user to control the

workbench angle based on application needs• L-shaped workbench


Projector arrays

• A large-volume display formed of a number of functionally identical projectors calibrated to display a tiled composite image of the virtual scene• Used to simultaneously satisfy the requirements of large image size, image brightness, and high resolution for wall-size displays• Requirements: Synchronization, Color uniformity and brightness•Example: PanoWall and Workwall


Blending the array images

• Very important for the image quality

• Approximately 25% of adjacent images are overlapped

• Increases luminosity in the overlapping zones

• Images coming from the multiple graphics computer is preprocessed


CAVE

• CAVE by Electronic Visualization Lab, Univ. of Illinois at Chicago, 1993

• Consists of 10-ft (~ 3m) cubical structure

• Four CRT projectors: The front, the left, the right sides, and the floor

• The projectors are synchronized to reduce flicker

• About 12 users wearing active glasses see a very convincing 3D scene


Domes

• Large, spherical displays that produce a 360o FOV surroundings

• As large as 400-500 viewers• Rely on multiple projectors arranged

radially around a semispherical rear-projection screen

• The number of projectors depends on the application and the dome dimensions

• Require special optics to pre-distort the image prior to projection on the curved screen

• V-Domes are used in high-tech theaters cost about 2 million dollars


• Differently polarizing filters are placed in front of each projector lens

• Users wear polarizing glasses where each lens only admits the light from the corresponding projector

Passive polarized stereo projector


Passive polarized stereo projector

• Use pairs of polarized projectors• Use inexpensive polarized glasses: About $1 each• “Passive” stereo glasses have lenses polarized

differently for the right and left eyes• Each eye sees the image component of the stereo

pair with matching polarization• The brain integrates the two images for the stereo

effect• Less expensive solution for large groups of people

looking at a stereo image than using active glasses


Sound Displays

• Computer interfaces that provide synthetic sound feedback to users interacting with the virtual world– The sound can be monaural (both ears hear the same

sound) or binaural (each ear hears a different sound)

• Play an important role in increasing the simulation realism by complementing the visual feedback provided by the graphics displays– When sound is added, the user’s interactivity, immersion,

and perceived image quality increase


Importance of Sound Important to create a sense of atmosphere Can greatly enhance feeling of presence Can be used to provide valuable depth cues, aiding navigation Enables the user to perceive events that occur outside the immediate field of view Highly immersive virtual reality simulations should have 3D sound in addition to graphics feedback

A virtual ball bouncing in a virtual room that is displayed on a CRT monitor: Simple monaural sound is sufficient A virtual ball bouncing in a virtual room that is displayed on a HMD: The simulation needs a device that provides binaural sound in order to localize the sound in 3D space relative to the user


Stereo vs. 3D sound

• Stereo sound on headphone– Seems to emanate inside the user’s head– Nor externalized as real sounds are– Sound source will follow the user’s head motion

• 3D sound on headphones or on speakers– Contain significant psychoacoustic information to alter the

user’s perception in believing that the recorded sound is actually coming from the user’s surroundings

– Sound source is localized in space– Direct sound from the source + sound bounced off the

walls, the floor, and the ceiling


The Human Auditory System

• Human perceive sound through vibrations arriving at the brain via the skeletal system or via the ear canal

• Elements used by the brain to measure the source location– – Intensity, frequency, and temporal cues present

in the sound perceived by the left and right ears


The Vertical-Polar Coordinate System

• Measure the sound source location: determined by three variables, azimuth, elevation, and range

• Azimuth angle– The angle between the nose and a plane containing the

source and the vertical axis z– -180o ≤ θ ≤ 180o

• Elevation angle– The angle between the horizontal plane by a line passing

through the source and the center of the head– -90o ≤ ψ ≤ 90o

• Range– The distance to the source measured along this line


Azimuth Cues

• The interaural time difference (ITD)– The difference in the arrival time of the sound at the

two ears– Sound has a fixed propagation velocity in air– Sound reaches first the ear that is closer the source– Maximum when the azimuth angle is 90o

– Zero when the azimuth angle is 0o, i.e., the source is directly in front of or directly behind the head


Azimuth Cues

• Interaural intensity difference (IID)– The difference in the intensity of sound reaching

the ears– The closest ear hears a sound with higher intensity

than reaching the distant ear– For lower frequencies (between 1.5kHz and

250Hz), the ITD dominates for azimuth localization as long as the source is not far from the user


Elevation Cues

• Ears are not simple holes• The path difference between the direct and pinna-

reflected sound changes with the elevation angle• Sound coming from a source located above the user’s

head has quite a different reflection path than sound coming from a source in front of the user: some frequencies are amplified and others are attenuated

• The pinna provides the primary cue for source elevation: the user’s face and shoulders geometry also influences the way the sound is reflected towards the outer ear


Range Cues

• Elements used to estimate range (or distance) from the user– – Prior knowledge of a given sound source– – Its perceived loudness

• A normally high-energy sound source– – When perceived as a faint sound, it is interpreted

as being distant, e.g., siren

• A normally faint sound– – When the sound is heard, it is interpreted as

coming from someone close, e.g., whisper


Range Cues

• Motion parallax– The change in sound source azimuth when the user is translating his or

her head– Large motion parallax: Indicates a source nearby (close to the user)– Little of no motion parallax: Indicates a source is far away from the user

• The ratio of sound coming directly from the source versus that which is first reflected off the user’s surroundings– Sound sources at distances greater than 1m produce a small ratio of

direct versus reverberated sound– The energy of the direct sound drops off with the square of the source

range– The energy of the reflected sound does not change much with range


3D Sound Modeling

• Assumes that the location of the sound source is known

• Needs a model of the corresponding sound reaching the inner ear

• The modeling task is complicated– Phenomenon multidimensionality– Individual anatomical variations– Partial understanding of the auditory system


HRIR & HRTF

• Place people inside a dome with spatially localized sound sources– Fit them with miniature microphones placed close to the

eardrum– The microphone output is stored and digitized as Head-

Related Impulse Response (HRIR) that contain all the physical cues used in source localization

– The corresponding Fourier transforms is called Head-Related Transfer Functions (HRTF)

– Each person has his own HRTF signature since no two persons have exactly the same outer ear and torso geometry


Headphone-Based 3D Sound

• Convolvotron– The first virtual 3D sound generator developed by Crystal

River Engineering– Use headphones and a set of PC-compatible dual cards– Sound can be synthesized based on one’s own HRTF– The spatial recognition rate dropped when use somebody

else’s HRTF– Not practical to design 3D sound processor customized for

each person’s HRTF– General approach is to use some generic HRTF and

accept the resulting inaccuracies


Speaker-Based 3D Sound

• Inexpensive PC 3D sound card• The PC loudspeakers are placed left and right of the monitor,

oriented parallel with it and facing the user• Assume that the user is facing the PC and within a “sweet spot”

zone Possible to create illusion of many more phantom speakers surrounding the user

• Unlike headphones, the speakers cannot isolate the sound for each ear the speaker output needs to be computed to assure cross-talk cancellation.

• TruSurround cards by SRS & SonicFury 3D by Videologic• Provide simplified 3D sound capabilities that are very much

appreciated by video game players• Provide the players the ability to hear the direction from which

an opponent approaches or shoots


Haptic Feedback

• Helps users achieve tactile identification of virtual objects in the environment and move these object to perform a task

• Visual + 3D audio + Haptic– Greatly improves simulation realism

• Haptic feedback groups the two modalities of touch and force feedback

• Designing good haptic feedback interfaces is a daunting task


Touch & Force Feedback

• Touch feedback– Conveys real-time information on contact surface

geometry, virtual object surface roughness, slippage, and temperature

– Does not actively resist the user’s contact motion and cannot stop the user from moving through virtual surfaces

• Force feedback– Provides real-time information on virtual object surface

compliance, object weight, and inertia– Actively resists the user’s contact motion and can stop it


Requirements in designing haptic feedback interfaces

• Safety– The forces he or she feels are real– These contact forces need to be large, but not large

enough to harm the user– Fail-safe: Users are not subject to accidents in case of

computer failure• Portability

– The need to provide sufficient force while still keeping the feedback hardware light and unintrusive

– Ease of use and installation at the simulation site• Self-contained

– Not supposed to require any special supporting construction, piping, or wiring


The Human Haptic System

• Input – Gathered by a multitude of tactile, proprioceptive,

and thermal sensors

• Output– Forces and torques resulting from muscular

exertion

• The system is not balanced– Humans perceive hapticly much faster than they

can respond


Tactile Feedback Interfaces

• Risky techniques– Electrotactile feedback: provides electric pulses to

the skin– Neuromuscular simulation: provides the signal

directly to the user’s primary cortex

• Commercially available techniques– Vibrotactile feedback: minimal cost and complexity– Temperature feedback


The Tactile Mouse

• The standard way of using the mouse– Requires that the user look at the screen all the time

• Tactile feedback– Remedy the requirement by adding another cue in

responses to the user’s actions– Characterize menu options, icons, or virtual objects hapticly

Feel differently

• iFeel Mouse by Logitech Co.– The addition of an electrical actuator can vibrate the mouse

outer shell– Provides vibration on the user’s palm


CyberTouch Glove

• By Immersion Corporation in 2002

• Provide vibrotactile feedback to the user– Able to provide feedback to individual fingers

• Most suitable for dexterous manipulation tasks– Contact is at the 5 fingertips and palm

• $15,000


Temperature Feedback Glove

• Allows users to detect thermal characteristics that can help identify an object material– Surface temperature, Thermal conductivity, & Diffusivity

• Thermal conductivity– Material with high conductivity (steel) feel cold– Material with low conductivity (wood) feel warm– Due to the heat flow from the finger to the object

• Temperature feedback– Increase the simulation realism– Useful when navigating in very cold/hot virtual worlds

• Displaced Temperature Sensing Systems (DTSS) by C&M Research


Force Feedback Interfaces

• Different aspects from the tactile feedback interfaces– The requirement to provide substantial forces to

stop user’s motion– Larger actuators, heavier structures, larger

complexity, and greater cost– Need to be grounded (rigidly attached) on some

supportive structures: Prevent slippage and potential accidents


Mechanical & Control Bandwidth

• Mechanical bandwidth– An important characteristic of force feedback interfaces– Represents the frequency of force and torque refreshes as

felt by the user

• Control bandwidth– Represents the frequency of command updates received by

the interface actuators– Always be larger than their mechanical bandwidthowing to

the inertia of the feedback structure

• The larger the force feedback interface– The larger output forces– The smaller mechanical bandwidth


• The WingMan Force 3D joystick– 3 DOF, 2 with force– $50

• The PHANToMArm– Stylus with 6 DOF, 3 with force

• The Haptic Master Arm– Larger work envelope

• The CyberGrasp Glove– Exoskeletons

• The CyberForce Glove– $56,000

Haptic Interfaces for Hands

cvvr lab.@ national dong hwa univ. 國立東華大學 vr output

Documents