BlockyTalky

BlockyTalky is a programming toolkit that empowers novices to make interactive, networked physical computing devices. Kids as young as ten have used it to build a wide range of projects, from networked cat feeders to computer music systems.

I started working on BlockyTalky in 2015 as a design-based researcher and developer at the Laboratory of Playful Computation (a research group within the University of Colorado Boulder). In 2016, I became the lead designer on the project.

Problem

Today's modern technologies are internet-connected and embedded with various sensors that gather data about our environments and bodies. Despite this, computer science education hasn't changed to address these new and exciting technologies. Most people do not learn about physical computing sensors and networking until advanced courses in college.

Goal

Educate and empower youth to learn about IoT and sensing technologies, and to build their own custom systems that leverage these technologies and are personally expressive and meaningful.

Generative Research

The first designs of BlockyTalky built on computer science education research that says block-based programming abstractions lower the barrier of entry to new programmers. We also built on connected learning and constructivist principles that argue students learn more effectively when they can connect their education to personally meaningful things in their lives.

Design

BlockyTalky involves a physical computing kit and a block-based programming language accessible from a web browser. Code that students write gets automatically uploaded to the physical BlockyTalky devices.

The BlockyTalky programming language is easy to learn. We designed it to foster computational thinking and problem solving by offering a tight match between the way novice programmers describe the interactive systems that they want to make and how they can express those ideas through code.

BlockyTalkys can communicate with one another. In this example, when the user presses the button on "Marceline," it sends a message to "Martian" which then plays music through the speaker.

BlockyTalky enables users to simply plug-and-play with a variety of sensors that don't require prior knowledge of circuitry or electronics.

BlockyTalky programming interface in a web browser.
Educational workshop evaluations

I have evaluated BlockyTalky in several informal K12 learning environments. The goal was to examine how students can learn computation using BlockyTalky, and if the connection to music and arts helps students identify more as programmers.

Methods:

  • design-based implementation research
  • surveys (identity, self-efficacy, opinions on BlockyTalky)
  • interviews (identity, discussion of networks and IoT)
  • discourse analysis
  • analysis of student code and documentation

Results:

  • music was an effective way to engage students with computing
  • identified areas of confusion regarding networks
  • identified technical bugs and issues

Co-design with teachers & evaluation

In 2016, I worked with 2 teachers to co-design in-school curriculums and new features for BlockyTalky. One was a middle school music teacher, and the other was a high school math and "computer power" teacher.

Music teacher's goals:

  • co-design BlockyTalky to better suit his music theory pedagogy
  • students work in groups to create musical compositions and performances
  • students create custom, interactive instruments for these performances
  • students develop more computational literacy and learn new music composition methods

Math/computer power teacher's goals:

  • students develop computational literacy and learn how to work with IoT and sensors
  • students learn the basics of computer networks
  • students create an interactive game or experience suited for a "carnival"
  • exposure to computer algorithms will align with math teacher's pedagogy

Our goals:

  • observe how students distribute work and collaborate with BlockyTalky
  • analyze impacts/effects of identity on the roles students engage with
  • evaluate the way students talk about networked systems and IoT before and after using BlockyTalky
  • generate designs how to make BlockyTalky more adaptive to different classroom needs

Co-designed feature where students can program motifs using scale degrees ("finger numbers").
Analysis and data visualization of network structures of students' projects (both classrooms).
Designs for adult artists and musicians

When I became the lead designer and developer of the BlockyTalky project, I began to investigate how we could rethink the design to make it support adult artists and musicians who wanted to build interactive creative systems.

To investigate this design space I took a mixed-method approach of naturalistic observation, surveys, focus groups, and one-on-one interviews. During this phase of research I identiified some ways new media artists were using interactive systems, such as visual jockeying (VJing), gestural music control, and wearable technology.

The problem was that developing these systems can be expensive and incredibly complicated to the novice programmer. I decided to design high-level abstractions that would let artists send data from BlockyTalky sensors to outside creative software using Open Sound Control (OSC). This enabled BlockyTalky to connect to programs like Max/MSP, Unity, Processing, and more.

Futher evaluation

I hosted a creative hackathon called Creative++. Monica Bolles and I designed the hackathon to be beginner-friendly and advertised the event to artists and musicians, as opposed to programmers and engineers. We introduced participants to the BlockyTalky environment, facilitated a project brainstorm session, led a group formation activity, and then groups spent twelve hours building a project that used physical controls and sensors to create some type of musical or artistic output.

Some of the collected data included:

Summary of findings:
The addition of OSC to BlockyTalky enabled groups to create interfaces to control professional artistic software like Max, Processing, and Unity instead of having to rely on the musical capabilities of the BlockyTalkys themselves. This ability to interoperate with more sophisticated software enabled the rapid construction and customization of live controllers for expression, demon- strating its potential beyond K-12 settings for use by adult creators. As expected, the projects that participants created still required at least one technical expert on the team. However, all participants were able to contribute to some level of the technical implementations of their projects. The non-technologists tended to assume the design-oriented roles, and their roles in the technical implementation primarily involved programming sensor-based or musical events within the BlockyTalky environment. This allowed novices to have some control over the interaction design of the system, despite not being able to contribute to work that required more traditional text-based programming.

Summary

BlockyTalky is a powerful product that supports personally-motivated learning and a connection to the arts. BlockyTalky lowers the barrier of entry for novice computer programmers by offering high-level abstractions that prevent syntax errors, as well as the ease of plugging and playing with new sensors.

Publications

BlockyTalky: New programmable tools to enable students’ learning of networks
Annie Kelly, Lila Finch, Monica Bolles, & R. Benjamin Shapiro
In Proceedings of the 2018 International Journal of Child-Computer Interaction

BlockyTalky: A Prototyping Toolkit for Digital Musical Interfaces
Annie Kelly, Monica Bolles, & R. Benjamin Shapiro
A workshop facilitated at the 2017 International Conference on New Interfaces for Musical Expression

BlockyTalky: Tangible Distributed Computer Music for Youth
R. Benjamin Shapiro, Annie Kelly, Matthew Ahrens, et al
Selected for the 2017 Computer Music Journal

BlockyTalky: A Physical and Distributed Computer Music Toolkit for Kids
R. Benjamin Shapiro, Annie Kelly, Matthew Ahrens, & Rebecca Fiebrink
In Proceedings of the 2016 International Conference on New Interfaces for Musical Expression