25/08/2012
1
Qualcomm Augmented Reality Lecture
Series – August 24, 2012
User Interfaces for Spatial AugmentedUser Interfaces for Spatial Augmented
Reality
Prof. Bruce H. Thomas
Director, Wearable Computer Lab
University of South Australia
The Team
o Computer Scientists
o Prof. Bruce H. Thomas
o PhD’s
o Michael Marner
o Designers
o SandyWalker
o Dr. Ivan Lee
o Dr. Robert Kong
o Dr. Ross Smith
o Dr. Stewart Von Itzstein
o Dr. Ash Doshi
o Jo Zucco
o Michael Marner
o Jo Zucco
o Shane Porter
o Markus Broecker
o Thuong Hoang
o James Walsh
o Matthew Adcock
o Steve Kelly
o Sean Pickersgill
o Peter Schumacher
o Adjuncts
o Dr. Wayne Piekarski
o Prof. Jun Park
o Prof. Rudi Vernik
o Guy Webber
o Ben Hughes
o Business Management
o Adele Flego
o Sebastien Herbert
25/08/2012
2
How Does Augmented Reality Technology
Fit in the Commercial World?
• Training
D i• Design
• Manufacturing
• Logistics
• Medicine
• Realityy
• Maintenance
How Does Augmented Reality Help?
• Just in time information
I it i li ti f d t• In situ visualisation of data
• Easier to understand information in context of the
real world
25/08/2012
3
What is Augmented Reality?
Like special effects in movies…
25/08/2012
4
Augmented Reality
• Registered virtual
information to physicalinformation to physical
world
• Provides extra
information
• Allows users to see the
unseeableunseeable
• Realtime
Presenting Augmented Reality
Information
Projectors / Spatial Displays Head Worn Displays Handheld Devicesj / p p y p y
25/08/2012
5
What Can You Do With AR?
• Embed manuals in the physical world
Vi li i t l t f t i t t f th• Visualise virtual artefacts in context of the
environment
• Status information in situ with physical object
• Simulate future capability
• Physically and virtual embody future concepts
Barriers to AR
• Applications have been too complex.
Vi li ti h b t biti• Visualisation have been too ambitious.
• Developers fight the current state of the art of
technology.
• Lack of good user interface technology.
25/08/2012
6
Spatial Augmented Reality
• Project perspective correct
graphics onto objects in thegraphics onto objects in the
real world
• Physical objects represented
as 3D virtual models
• Can project onto movable
objects by adding a trackingobjects by adding a tracking
system
Spatial Augmented Reality in five
easy steps
Ramesh Raskar, Greg Welch, Kok‐lim
Low Deepak Bandyopadhyay
Shader Lamps:
Animating Real Objects with Image‐
Based Illumination
25/08/2012
7
Step One
Step Two
25/08/2012
8
Step Three
Step Four
Six Degrees of Freedom
(X Y Z Ψ Θ Φ)(X, Y, Z, Ψ, Θ, Φ)
25/08/2012
9
Step Five
Challengers for SAR User Interfaces
• Traditional UI technology does not map onto SAR
N k b d– No mouse or keyboard
• Many virtual environment technologies do not map
well onto SAR
– All virtual information must be displayed on a
physical surface, . no floating menus.
• Need to rethink the user interface!
25/08/2012
10
Opportunities for SAR User Interfaces
• Users do not have to wear a head mounted display
Th i h ti t f th di l f• The passive haptic nature of the display surfaces
allows for better understanding of the
physical/virtual visualisation.
– Users can touch and interact with the display
surface
N ll ll b i• Naturally supports collaboration
– all users are able to see the same virtual
information
Physical-Virtual Tools for Spatial
Augmented Reality InteractionAugmented Reality Interaction
Michael R. Marner
25/08/2012
11
Augmented Foam Sculpting for
Capturing 3D Models
• How can we make this process better?
Augmented Foam Sculpting for
Capturing 3D Models
25/08/2012
12
Evaluation
- Quality of Virtual Models
• Augmented Foam Sculpting compared to Polhemus
FastSCANFastSCAN
• 684 triangles vs. 81,925
– Faces only added where cut occurs
• Accuracy currently limited to tracking system
Research Question
• How should users
interact with largeinteract with large
SAR systems?
• Where does the
keyboard and mouse
go?
25/08/2012
13
Physical-Virtual Tools
• User interaction supported
by physical toolsby physical tools
• Aim to take advantage of
existing skills
• User feedback projected
onto tools. Can overload
tools with several tasks by
h i j ichanging projections
• No need for separate
screens, HUD, etc.
(A very brief overview of)
Related Work
• Graspable & Tangible User Interfaces
Fitzmaurice et al 1995 Ullmer& Ishii 1997– Fitzmaurice et al. 1995, Ullmer& Ishii 1997
• Props used as handles to virtual objects
– Spray Modeling (Jung et al. 2004)
– Surface Drawing (Schwerdtfeger et al. 2008)
• VR input devices
– Virtual Tricorder (Wloka& Greenfield 1995)
– Personal Interaction Panel (Szalavri&Gervautz 1997)
• Shader Lamps (Raskar et al. 2001)
25/08/2012
14
Digital Airbrushing with a Virtual
Stencil
Digital Airbrushing with a Virtual
Stencil
• Natural, two handed interaction
T k d t f i ti i b hi kill– Takes advantage of existing airbrushing skills
• Work can be saved and modified later
• Virtual stencil is more flexible
– Any shape can be used
25/08/2012
15
Tool Virtualization and Spatial
Augmented RealityAugmented Reality
Michael R. Marner Bruce H. Thomas
@
The Within-System Tool
Virtualisation Continuum
25/08/2012
16
The Within-System Tool
Virtualisation Continuum
• helps designers decide on the number of tools
required and the physical nature of these toolsrequired, and the physical nature of these tools
• A task is the physical interaction required to achieve
a goal (. drawing a line)
• An attribute is a user changeable parameter for a
task (. line color)
O l d l i h k h h h• Overload tools with tasks that have the same or
similar physical interactions
The Within-System Tool
Virtualisation Continuum
• As a tool’s position in the y‐axis increases, should
consider adding another tool with the same physicalconsider adding another tool with the same physical
characteristics
• As a tool’s position in the x‐axis increases, should
consider adding another tool with physical
characteristics customized for a subset of the tasks
25/08/2012
17
The Inter-System Tool Virtualisation
Continuum
The Inter-System Tool Virtualisation
Continuum
• Used to compare the relative complexity of user
interfaces of different systemsinterfaces of different systems
• The complexity of user interface increases as the
system moves away from the origin of the graph
• Systems in the lower left of the IS‐TVC would have
less complex user interfaces
25/08/2012
18
Example Tools
Ray Gun Tool
• Digital Airbrushing
C l i t fl– Color, paint flow,
hardness, and spray
angle attributes
• Virtual Laser Pointer
– Color attribute
• Stencil Creation
• Command Entry
• Placed (4,4) on WC‐TVC
25/08/2012
19
Stylus Tool
• Annotation
P l– Pen color
• Stencil Creation
• Command Entry
• Placed (3,1) on WC‐TVC
Foam Sculpting Tool
• Highly specialised tool
A t d F S l ti• Augmented Foam Sculpting
– No attributes
• Placed (1,0) on WC‐TVC
25/08/2012
20
View-dependent Rendering andView-dependent Rendering and
Depth-perception in SAR
Markus Broecker
View-dependent Rendering
• Projecting onto surfaces
Si l d l l• Simple models ‐ complex
CAD data
• Holes, edges, silhouettes
• Virtual space
• View‐dependent/
”perspective correct”
rendering
25/08/2012
21
Three Things
• How well can we trick the user in believing this
illusion?illusion?
• View‐dependent Rendering
• Depth Cues/Perception
• User study
(View-dependent) Rendering basics
T diti l AR/VR• Traditional AR/VR:
– View position is
implicitly known
– ‘Window’ into VR
world
Viewer
– Pinhole camera
• Fishtank VR
• SAR?
Screen
Virtual World
25/08/2012
22
Rendering in SAR
Viewfrustum
Projector
VR ModelProp
But where’s the user?
View-dependent Rendering
Implementation
Vi iti !• View position !=
Render position
• Tracking
• Projected texturing
• Two‐passes:
– Render virtual space
from user’s position
– Project as texture
Virtual Space SAR scene
25/08/2012
23
View-dependent Rendering Results
Depth Cues/Perception
• Image is a 2D ‘plane’
• Depth and distance are inferred
25/08/2012
24
Claude Lorrain ‐ Seaport with the Embarkation of St. Ursula, 1641
Usable Depth Cues for SAR
P ti Grid• Perspective
• Occlusion
• Shading
• Shadows
• Texturing
‘automatically’
provided by CG
OpenGL
settings
✤ Grid
✤ Selection Ray
✤ Blur
artificial
depth cues
g
• Parallax
• ‘Thickness’
Head‐Tracking
Virtual‐space
25/08/2012
25
Measuring Depth Perception
D kt SAR• Desktop SAR
• ‘Empty’ box
• Selection task
• Ground truth
User Study (ongoing)
• 2 Boxes
M l ti• Many selections
• Combinations of depth cues
• Hypotheses:
• Parallax is strongest depth cueg p
• View‐dependent Rendering at least ‘as good as the
real thing’
25/08/2012
26
Results (shakycam)
QuickTime™ and a
decompressor
are needed to see this picture.
TAM
Ad-hoc Tangible User Interaction
James Walsh
25/08/2012
27
Inspiration
• What is TAM?
– Architecture to enable ad‐hoc definition of TangibleArchitecture to enable ad hoc definition of Tangible
UIs (TUIs)
• Distinct separation of those who develop TUIs and those
who use them
– Individual development and usage stages – doesn’t
fl t l ldreflect real world use
• Why can’t we just walk up to a system, pull out an object,
and start interacting using it?
Tangible UIs
• Marble answering machine (Ishii, 2008a)
– Physically instantiated virtual elements
• Dolls head prop (Hinckley et al., 1994)
– Surgical visualization through spatial relationships
• The Proximity Toolkit (Marquardt et. al, 2011)
– Toolkit for abstracting proxemic relationships to
ti t
Dolls head
(Hinckley et al.,
1994)
programmatic events
• Physical‐Virtual tools (Fjeld et al., 2002)
– Physical tools to support virtual interactions
PVT Spray Painting
(Marner et al., 2009)
25/08/2012
28
UI Theory
• CLI > GUI > NUI > Continuous NUI
(Petersen and Stricker, 2009)
• Model‐View Controller architecture used
to enable GUIs
(Olsen, 1998)
– Needs to be modified for TUIs (Ullmer
2002)
• Current evaluation frameworks don’t
support adaptive UIs
(Stary and Totter, 1997)
Interface Design
• Programming by Demonstration vs. Programming by Example
(Halbert 1984)
– “do what I did” vs. “do what I mean”
• Need to support trial‐and‐error exploration of interfaces
(Sharlin et al., 2004)
• TUI equivalents of GUI concepts (Ullmer and Ishii, 1997)
• Time multiplexed vs. space multiplexed
– Specificity vs. Generality
– Strive for 1:1 ratio (Sharlin et al., 2004)
(Ullmer and Ishii, 1997)
25/08/2012
29
Research Question
“How can we introduce previously unknown
t ibl UI d th t ff ti l i t ttangible UIs and use them to effectively interact
with the system?”
Need to support range of devices ‐ completely passive
to organic
Implementation
• Gestures used for navigation
– Wanted to avoid translating GUI into tangible realm
– Allows entire system space to be ‘blank’ for user
definition
• Kinect used for object detection and enabling touch
interactions
• 6DOF IR tracking system used to track objects• 6DOF IR tracking system used to track objects
• Introduce Objects > Define Properties > Define
Interactions > Interact
25/08/2012
30
Implementation
Using the System
25/08/2012
31
Evaluation
• Participants watched video showing a single
interaction being createdinteraction being created
• Users could only watch the video once and had no
additional materials aside from a ‘gesture guide’
• Asked to create 3 simple ‘war table’ type scenarios to
demonstrate system functionality
Results
• All participants completed all three stages
• Majority of users successfully completed the scenario in
their first attempt
– Indicates their expectations of the system matched
the system’s functionality
• Some participants confused over use of groups
• Gesture‐based navigation was terrible
25/08/2012
32
UniSA Holodeck:
The World’s Largest SpatialThe World s Largest Spatial
Augmented Reality Research Center
How do you design…
Appliances?pp
25/08/2012
33
How do you design…
Appliances? Operating Theatres?pp p g
How do you design…
Appliances? Operating Theatres? Command and Control Rooms?pp p g
25/08/2012
34
The Problem
Design Space
The Problem
Virtual
Designs
Design Space
25/08/2012
35
The Problem
Virtual
Designs
Design Space
The Problem
Virtual
Designs
UniSA
Holodeck
Design Space
25/08/2012
36
UniSA Holodeck
• 14m x x 4m
44 P j t• 44 Projectors
• High Performance
Computing
• 40 Cameras
• Wide area tracking
• Automatic configuration
• External load‐in
Command and Control Room
Design
25/08/2012
37
Spatial Augmented Reality for
Design
• Command and control centres on submarines and
shipsships
• Command centres for mining and gas operations
• Rapidly deployable command and control centres in
the field
• Hospital operating theatres
• Any interior design problem
Benefits
• Natural manner for viewing designs
St k h ld b i f j d i i• Stakeholder buy‐in for major decisions
• Early ergonomic measurements
• Interactive simulations of new technology
25/08/2012
38
Current Industrial Projects
• GM Holden
M f t i d bl– Manufacturing and assembly
• Intel
– Maintenance tasks
• Jumbo Vision
– Design of control roomsg
