Audiovisual Playground

As electronic music has made its way into the mainstream, the popularity of music production tools for both professional and amateur use has grown. Although the users of music technology are becoming more prevalent and diverse, there have been very few innovations in their user interfaces. The onset of high-quality, consumer virtual reality has allowed for the development of unique user interfaces that are able to make use of an interactive 3D space, as opposed to a standard 2D interface. Audiovisual Playground is a 3D virtual reality music application my collaborators and I developed for the HTC VIVE. Audiovisual Playground builds off of a programmable music device called a sequencer, and extends its design metaphors to a large-scale, 3D environment.

Icospheres are the objects users interact with to control which instruments trigger on different beats.
An icosphere. An icosphere. An icosphere.
A non-glowing icosphere indicates that a music sequencer beat is not active.
A glowing, but transparent icosphere indicates where the "first beat" is on a given instrument pedal (but it is not active until clicked).
A glowing and opaque icosphere means the user has activated that beat and will hear its audio playback.
Problem:

Many virtual reality applications simply recreate 2D interfaces, as opposed to completely rethinking how these interfaces can be designed to leverage the affordances of embodied user interaction in a 3-dimensional space.

Goal:

Design a novel interface of a 3D virtual reality music sequencer that provides an embodied and immersive music-making experience.

Video capture of a user playing Audiovisual Playground.
Instructions for users.
Design

We designed the interface to encourage user's to walk around the space to interact with different instruments. We relied on UI design principles for the HTC Vive, such as raycasting from the controller to show users where they are pointing, and updating objects to have a glow around them when selected.

Normal representation of a music sequencer (left), an aerial view we imagined of a 3D music sequencer (right).
Evaluation:

We evaluated AudioVisual Playground by running in situ usability tests with ten users. We used a mixed-methods approach. We collected quantitative information such as logs of users' interactions and the amount of time spent performing different actions. We also collected qualitative data regarding users' emotional and cognitive experiences.

In situ testing of Audiovisual Playground.
Publications

Audiovisual Playground: A Music Sequencing Tool for 3D Virtual Worlds
Annie Kelly & Kristofer Klipfel
In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

Other media

Description of project on the ATLAS Institute website
Mention of project on CU Boulder's AR/VR club page