caadfutures2017: integration of a structure from motion into virtual and augmented reality for...
Post on 24-Jan-2018
603 Views
Preview:
TRANSCRIPT
Integration of a Structure from Motion
into Virtual and Augmented Reality
for Architectural and Urban Simulation
July 13 2017
Tomohiro Fukuda1 Hideki Nada2 Haruo Adachi2
Shunta Shimizu3 Chikako Takei3 Yusuke Sato1
Nobuyoshi Yabuki1 Ali Motamedi1
1
Demonstrated in Real Architectural and Urban Projects
1 Osaka University, Osaka, Japan
2 Sakaiminato City Office, Tottori, Japan
3 Forum8 Co., Ltd., Tokyo, Japan
Contents1. Introduction
2. SfM-based Modeling for VR
3. Using Camera Parameters of SfM for Marker-less AR
4. Conclusions
2
Virtual and Augmented Reality
Constructs 3D past and future virtual spaces with real-time andinteractive operation.
Considerable amount of time and expense is entailed on anurban scale to construct and integrate 3D objects for all virtualspaces.
3
RealEnv.
AugmentedReality (AR)
AugmentedVirtuality (AV)
Virtual Env. = VR
Mixed Reality (MR)
Simplified representation of a "virtuality continuum“ by Milgram and Kishino (1994)
VR Azuchi Castle prj.
CAAD Futures 2015, Sao Paulo
Housing Design prj.
eCAADe 2015, Vienna
Virtual and Augmented Reality
Overlays 3D virtual objects on real world images
Can help visualize full-scale design projects on a planning site
Must track the geometric registration between live video and 3Dobjects to accurately render the outdoor AR
4
RealEnv.
AugmentedReality (AR)
AugmentedVirtuality (AV)
Virtual Env. = VR
Mixed Reality (MR)
Simplified representation of a "virtuality continuum“ by Milgram and Kishino (1994)
Sensor-based AReCAADe2006, Volos
Marker-based AR2015 Int’l WS on Computing in Civil Engineering, Texas
17 22 27℃
SfM (Structure from Motion)
Estimates 3D structures from 2D sequential images that are coupled to local motion signals
Requires only familiar equipment (Smart phones), free or low-cost software
5Google Maps
Target Building
Shooting routes and directions
Camera parameter
Point clouds点群
Dense point clouds
1,335,958 points Camera Parameter3,194,585 faces
1,598568 vertices
Objective
Integration of SfM into VR and AR:
A 3D modeling method by SfM are explored for real-time
rendering to realize less modeling cost in large-scale VR
Reduce a large number of generated point clouds and meshes
for the frame rate of real-time rendering
Camera parameters of SfM are explored for a marker-less AR
to realize registration and tracking accuracy to visualize full-
scaled architectural design projects
Apply SfM-based AR to real architectural design projects to
examine the uses and limitations
6
Contents1. Introduction
2. SfM-based Modeling for VR
3. Using Camera Parameters of SfM for Marker-less AR
4. Conclusions
7
Urban Design Project:
Mizuki Shigeru Road Renewal
4
Challenges Aging
Barrier-free
Schematic renovation plan
Creating a hospitable and entertaining road
that everyone would like to visit
Shigeru Mizuki MuseumJR Sakaiminato st.800m
水木マンガの世界
森にすむ妖怪たち
神仏・吉凶を司る妖怪たち 身近なところにひそむ妖怪たち
川や水辺 野や里 山 海辺 町家に棲む妖怪たち
Creation of a one-way and S-shaped roadway
Creation of a wider sidewalk space
Re-accommodation of the bronze statues within the new
zone concept
Improvement of cityscape with a sense of unity
Night landscape design
15
16
VR production
Challenges 3D digital modeling of 153 bronze statues
Possess different shapes and included complicated shapes with holes
Conventional method Modeling using 3DCG software demands enormous time and cost
resources
17
Construction design study and public meeting to build
consensus
Representation of a Human’s Motion and Direction by 2D images in VR
(Fukuda et al., Int’l J. of Arch. Comput. 7(2), 2009)
Photography for SfMConditions
Shoot on cloudy days To generate a shadow-less texture at 3D
modeling by SfM to ensure optical consistency
between VR and 3D objects by SfM
Shoot at times of less tourists Not to shoot noise elements
Role sharing
Sakaiminato city office staffs took 10,000
photos of 153 statues in a month VR experts worked away from the project site
18
Photographs for SfM
19
SfM-based Statues Modeling
20
High4286 polygons
Medium1025 polygons
Low298 polygons
Software: PhotoScan, Remake,UC-win/Road SfM plug-in
Shooting time: 10-20 min./statue
Production time: Automatically generated 30 min. Editing 30min.
SfM-based Statues Modeling
153 bronze statues are ranked
into 3 categories:
High 27 Med 46 Low 80
21
OS Windows7 Professional 64-bit
CPU Intel Corei7-3770 3.40GHz
RAM 16GB
GPU NVIDIA GeForce GTX 670 VRAM 4GB)
Res. 1920x1080
100 statues
Accuracy fps
High 16.4
Med. 24.1
Low 27.1
When all the high-accuracy statues are
installed, realizing real-time rendering is
difficult.
Results: 3D model by SfM
22
河童
ねずみ男 ガラッパ 小豆洗い
鬼太郎と目玉おやじ
かみきりと一反木綿
袖引小僧龍水木しげる先生 執筆中
(N=27) (N=46) (N=80)
SfM-based Roadside Building Modeling
23
Conventional method
Polygons 19
Vertices 48
120 buildings along the roadside▶ Difficult to realize real-time
rendering using SfM-based 3D model
SfM (e.g. PhotoScan)
Polygons 2.4 million
Vertices 1.2 million
SfM-based Roadside Building Modeling
24
Conventional method
Editing work still involves a considerable amount of time.
The roadside buildings were finally created by using the
conventional method.
SfM-Photoscan SfM-Remake
Lack of content: Elements to be modeled are not
modeled by SfM (roof)
Inadequate content: Geometry is not correct
although it is modeled by SfM (signs, arcade)
Surplus content: Unnecessary objects are
modeled (wire contains sky elements)
25
3D modelNumber of polygons 965,472
Number of vertices 929,325
Texture
Number of textures 885
Number of texture pixels 65,530,368
Data amount 1.5 GB
VRAM use when running VR application 667.4 MB
TotalData amount 1.35 GB
Frame rate 30-50 fps
26
27
28水木しげるロードリニューアルイメージ映像
Contents1. Introduction
2. SfM-based Modeling for VR
3. Using Camera Parameters of SfM for Marker-less AR
4. Conclusions
29
Marker-less AR: System Flow
30
Marker-less AR: System Flow
31
Photographs from viewpoints
Marker-less AR: System Flow
32
Store position and orientation data of all camera
3D point cloud reconstructed by SfM
Marker-less AR: System Flow
33
Define relative coordinates of 3D
design objects using SfM model
Marker-less AR: System Flow
34
Start video capture (OpenCV 2.4)
Marker-less AR: System Flow
35
Extract features of the live video image
Compare them with features in DB
using SURF (Speeded-up Robust Features)
Extract the camera’s position and
orientation of the most similar image
Marker-less AR: System Flow
36
Render 3D virtual objects precisely in AR
using referred position and orientation data
Marker-less AR: System Flow
37
Motion vectors for tracking are calculated by the optical flow technique.
Positions of points on the screen and the corresponding world coordinates of those
points are calculated in real time.
Application in Building Design prj.
38
To study the facade and approach of the new office building at the preliminary design stage (Jan – Mar 2016)
Demolish the existing office building and build a new three-story office building in Osaka, Japan
0 20m
Interview survey
Advantage
The size of the building could be intuitively comprehended at
the construction site
It was possible to obtain deeper communication in the
preliminary design stage.
Disadvantage
Difficulty of viewing the exterior as well as the building model
through the tablet LCD display.
Walking freely by using a video see-through type HMD is difficult
because the surrounding environment is not adequately visible.
Using an optical see-through HMD, a problem exists wherein it is
difficult to view the AR due to the effect of sunlight.
39
Interview survey
Although the size of the rendered 3D model is accurate, an
opinion persisted that this was not possible.
The validity of the size of the designed building can be
explained by comparing it with the size of the adjacent
building in the experiment.
40
Contents1. Introduction
2. SfM-based Modeling for VR
3. Using Camera Parameters of SfM for Marker-less AR
4. Conclusions
41
Conclusions Integrating SfM into VR and AR is demonstrated for
architectural and urban visual simulations.
A modeling method using SfM is developed to model a number
of 3D objects under real-time rendering conditions. The total
data amount of bronze statues is suppressed by setting three
reproduction ranks. A large-scale VR application can be
constructed with decreasing the modeling cost.
An AR system using SfM camera parameters for registration is
developed to visualize design projects. Design stakeholders
could study the size and facade of the office building by using
the full-scale displayed AR at the planning site. It is possible to
obtain a more constructive communication among
stakeholders.
A future study should be examined further integration into VR
and AR based on the technology to restore a 3D model in real
time from camera images (e.g., Simultaneous Localization and Mapping).
42
Acknowledgements The research was partly supported by JSPS KAKENHI Grant
Number JP25350010 and JP26-04368, and joint research
funding between Sakaiminato city office and Osaka University.
We would also like to take the opportunity to thank Kyokuyo
Electric Co., Ltd. and atelier DoN to apply our proposed AR
system to the building design project.
We thank all the participants for their generous assistance in
conducting projects and experiments.
4343Mizuki Shigeru Road (Jul 2017) Kyokuyo Electric Co. Ltd. New head office (Jul 2017)
top related