All VR’s a Stage: The Aesthetics of Immersive Mixed Reality Theater

Virtual Reality presents great promise as a storytelling medium, but rarely delivers on that promise because it is often approached as an offshoot of cinema. Virtual Reality as a narrative medium has much more in common with theater, using multi-modal narrative in a three dimensional space to tell the story. The possibility of multiple users sharing a virtual space simultaneously creates the opportunity for live performance, with one or more performers moving among and around the audience -- immersive theater within a computer-generated setting. This paper examines the aesthetics of this new space for digital performance. 

We introduce a new type of performance activity, “Immersive Mixed Reality Theatre” (IMRT), which promises exciting possibilities for participatory immersive digital narratives. To explore the potential aesthetics of IMRT we created Holojam In Wonderland (2017), a short play inspired by the work of Lewis Carroll. It was built on the Holojam platform developed by the NYU Future Reality Lab, which enables both performers and audience to walk around with untethered VR headsets within the same room. 

Sharing the same physical space is part of the enchantment of live theatre, where the audience feels they could touch the actors; this proximity with the bodies and voices of the performers creates a visceral connection with the audience. This is the sense of presence that is often elusive in Virtual Reality. We wanted to transpose the excitement of live theatre to a visually expansive experience mediated by computer graphics, allowing a live co-located audience to experience a performed narrative via a shared immersive digital world.

Our work differs from other mediated experiences, including other types of Virtual Reality, in which the viewer is physically isolated. Rather, IMRT is a new type of theatrical experience that invites shared immersion by audience and performers within a digital mediated space, while retaining physical proximity. 

IMRT presents specific design challenges. Because all participants are represented as computer-generated avatars, the aesthetics of IMRT inherit features from animated film and puppetry. As IMRT occurs in a physical space, the performance and movements must still be connected to its physical components, to allow all participants to remain aware of each other and of the space. In addition, an experience must consider how the audience transitions between the physical and digital worlds.

We have also explored the additional affordances of virtual reality by experimenting with features that are not possible in traditional theater. Individual head-mounted displays allow for personalized teleprompters, visible only to individual performers reading their cues. Computer generated imagery affords many effects, such as changing the scale of objects and characters. Having a live performer become a giant or lilliputian allowed us to create a powerful sense of wonder, which informed our decision to use Alice in Wonderland as the first text to appropriate and adapt. As an exploration, Holojam in Wonderland uses Alice as a guide into a new and exciting dream world, in this case the land of immersive mixed reality theater.