An augmented reality toolkit for prototyping live artistic controllers

ARcadia is a toolkit for bringing paper prototypes to life through the use of marker-based Augmented Reality. It enables people to create tangible interfaces for real-time creative expression without the use of embedded sensors or electronics.

The left image shows what a typical table-top ARcadia setup looks like. The right image shows a user interacting with their physical ARcadia interface elements.

Users can construct interfaces using cardboard, paper, and other craft materials and stick fiducial markers on it to enable tracking via a webcam. Users can then program different mappings between markers and events to add interactivity to their projects. After the crafting and programming phase, a user can play with their interface live – meaning their interactions with the interface will cause events to occur in real-time. Examples of possible projects include drum machines, pianos, games, etc with only using cardboard, fiducial markers, and a camera.

A piece of code that a user wrote for a simple ARcadia project.

I developed ARcadia in collaboration with the Microsoft MakeCode team while I was an intern there. Since my internship ending, I have continued to conduct user testing and iterate on the design of ARcadia using design based research methods. In the process of developing ARcadia I have used best UI/UX practices as well as standards that Microsoft Research developed.

Using physical ARcadia elements to control a YouTube video.

Annie Kelly, R. Benjamin Shapiro, Jonathan de Halleux, and Thomas Ball. 2018. ARcadia: A Rapid Prototyping Platform for Real-time Tangible Interfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). Association for Computing Machinery, New York, NY, USA, Paper 409, 1–8. DOI: