Art der Publikation: Beitrag in Sammelwerk

Scene Responsiveness for Visuotactile Illusions in Mixed Reality (Best Paper Award)

Autor(en):

Kari, Mohamed; Schütte, Reinhard; Sodhi, Raj

Titel des Sammelbands:

Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

Verlag:

Association for Computing Machinery

Ort(e):

New York, NY, USA

Veröffentlichung:

2023

ISBN:

9798400701320

Schlagworte:

Mixed reality, situated computing, spatial computing;

Digital Object Identifier (DOI):

doi:10.1145/3586183.3606825

Zitation:

Download BibTeX

Kurzfassung

Manipulating their environment is one of the fundamental actions that humans, and actors more generally, perform. Yet, today’s mixed reality systems enable us to situate virtual content in the physical scene but fall short of expanding the visual illusion to believable environment manipulations. In this paper, we present the concept and system of Scene Responsiveness, the visual illusion that virtual actions affect the physical scene. Using co-aligned digital twins for coherence-preserving just-in-time virtualization of physical objects in the environment, Scene Responsiveness allows actors to seemingly manipulate physical objects as if they were virtual. Based on Scene Responsiveness, we propose two general types of end to-end illusionary experiences that ensure visuotactile consistency through the presented techniques of object elusiveness and object rephysicalization. We demonstrate how our Daydreaming illusion enables virtual characters to enter the scene through a physically closed door and vandalize the physical scene, or users to enchant and summon far-away physical objects. In a user evaluation of our Copperfield illusion, we found that Scene Responsiveness can be rendered so convincingly that it lends itself to magic tricks. We present our system architecture and conclude by discussing the implications of scene-responsive mixed reality for gaming and telepresence.