I led the design and research of ARcadia at the Laboratory for Playful Computation in Boulder, CO, and at Microsoft Research in Redmond, WA.
ARcadia brings paper prototypes to life through the use of marker-based augmented reality (AR). With ARcadia, people can turn real-world tangible objects into controllers for music, IoT devices, lights, and more. ARcadia empowers novice technology designers to rapidly construct and iterate on these interactive prototypes.
Designing custom tangible interfaces requires complex knowledge of programming and electronics. It is also expensive, and can be challenging for people (even experts) to iterate on their designs.
Design a low-cost, beginner-friendly programming toolkit that enables users to turn any physical object into an interactive, tangible user interface.
ARcadia was designed for use in K-12 informal and formal computer science education settings, and for people of any age that want to turn objects into interactive controllers without prior programming experience.
ARcadia enables users to any physical object interactive by placing a trackable, unique barcode on it. A user can then point a webcam or phone camera at these trackable objects, which are detected by the system. In the ARcadia programming editor, users can use code blocks to develop event-driven programs that define how their interfaces behave. For example, they can write a program that says, "when this object is moved to the left, play a snare drum."
By allowing users to develop programs using high-level code abstractions, and to only require a computer and basic low-cost materials — users can rapidly prototype and design novel, tangible interfaces for creative expression.
In August 2018, I tested ARcadia with 120 high school aged girls by facilitating an augmented reality music-making workshop. I guided workshop participants through a few introductory exercises and then let them work on projects of their choosing for one hour. I collected their project code, photographs and videos of their projects through various levels of completion, and collected participants' responses for things they "liked about ARcadia," and things they "wish ARcadia could do."
Based on feedback I collected from participants, I identified a weakness in the ARcadia toolkit. When I say "weakness" I am not referring to performance issues or bugs, but rather the way ARcadia was designed to be used. Participants wanted to use it for projects beyond the simple music features that were available in the browser. They wanted to use it to create custom controls for games, to control lighting in their houses, and more. In other words, ARcadia needed to be flexible enough to connect to many different applications through the internet of things (IoT).
To address this feedback, I designed ARcadia to contain begnner-friendly programming constructs that would enable users to connect their interfaces to other applications. To perform contextual inquiry, I interviewed designers and performing artists to gain an understanding of what interesting use cases might be, using semi-structured interviews and questionnaires. I found that in the performing arts and IoT community, it is common for creators and hackers to route input and output between multiple applications using WebRTC and OSC (two technologies that enable networked communication). I added blocks that networked over WebRTC and OSC. Below are three examples: ARcadia used to control lighting, ARcadia used as an effects pedal for a keyboard, and ARcadia used to scrub a YouTube video to alter audio and visual effects.
To test the new networking features of ARcadia I tested with users in situ, as well as conducted usability testing in the field at pop-up demo booths. In addition, to push the capabilities of ARcadia to the max I along with other performers testing using ARcadia as musical controllers, video game controllers, visual jockeying controllers, home automation controllers, etc.
In my evaluation I identified that some of the labels on the networking blocks were confusing, and there were too many steps required to initiate connection between devices. To address this, I redesigned the blocks such that devices could connect by name instead of having to use IP addresses or long strings of numbers and letters.
Overall, ARcadia makes it much easier for non-programmers to design multimedia, interactive, networked systems.
ARcadia has been implemented as a computer science education tool in international classrooms and learning environments. I have published my ARcadia design work in the 2018 CHI Conference on Human Factors in Computing Systems, and ARcadia has also been referenced by others UI/UX designers and researchers.
ARcadia: A Rapid Prototyping Platform for Real-time Tangible Interfaces
Annie Kelly, R. Benjamin Shapiro, Peli de Halleux, & Thomas Ball
In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
On list of Microsoft MakeCode custom editors
Project description on Laboratory for Playful Computation website
Mention of research publication of Microsoft Research's website
Mention of project and my overarching thesis research on the ATLAS Institute's website