Today's modern technologies are internet-connected and embedded with various sensors that gather data about our environments and bodies. Despite this, computer science education hasn't changed to address these new and exciting technologies. Most people do not learn about physical computing sensors and networking until advanced courses in college.
Educate and empower youth to learn about IoT and sensing technologies, and to build their own custom systems that leverage these technologies and are personally expressive and meaningful.
The first designs of BlockyTalky built on computer science education research that says block-based programming abstractions lower the barrier of entry to new programmers. We also built on connected learning and constructivist principles that argue students learn more effectively when they can connect their education to personally meaningful things in their lives.
BlockyTalky involves a physical computing kit and a block-based programming language accessible from a web browser. Code that students write gets automatically uploaded to the physical BlockyTalky devices.
The BlockyTalky programming language is easy to learn. We designed it to foster computational thinking and problem solving by offering a tight match between the way novice programmers describe the interactive systems that they want to make and how they can express those ideas through code.
BlockyTalky enables users to simply plug-and-play with a variety of sensors that don't require prior knowledge of circuitry or electronics.
I have evaluated BlockyTalky in several informal K12 learning environments. The goal was to examine how students can learn computation using BlockyTalky, and if the connection to music and arts helps students identify more as programmers.
In 2016, I worked with 2 teachers to co-design in-school curriculums and new features for BlockyTalky. One was a middle school music teacher, and the other was a high school math and "computer power" teacher.
Music teacher's goals:
Math/computer power teacher's goals:
When I became the lead designer and developer of the BlockyTalky project, I began to investigate how we could rethink the design to make it support adult artists and musicians who wanted to
build interactive creative systems.
To investigate this design space I took a mixed-method approach of naturalistic observation, surveys, focus groups, and one-on-one interviews. During this phase of research I identiified some ways new media artists were using interactive systems, such as visual jockeying (VJing), gestural music control, and wearable technology.
The problem was that developing these systems can be expensive and incredibly complicated to the novice programmer. I decided to design high-level abstractions that would let artists send data from BlockyTalky sensors to outside creative software using Open Sound Control (OSC). This enabled BlockyTalky to connect to programs like Max/MSP, Unity, Processing, and more.
I hosted a creative hackathon called Creative++. Monica Bolles and I designed the hackathon to be beginner-friendly and advertised the event to artists and musicians, as opposed to programmers and engineers. We introduced participants to the BlockyTalky environment, facilitated a project brainstorm session, led a group formation activity, and then groups spent twelve hours building a project that used physical controls and sensors to create some type of musical or artistic output.
Some of the collected data included:
Summary of findings:
The addition of OSC to BlockyTalky enabled groups to create interfaces to control professional artistic software like Max, Processing, and Unity instead of having to rely on the musical capabilities of the BlockyTalkys themselves. This ability to interoperate with more sophisticated software enabled the rapid construction and customization of live controllers for expression, demon- strating its potential beyond K-12 settings for use by adult creators. As expected, the projects that participants created still required at least one technical expert on the team. However, all participants were able to contribute to some level of the technical implementations of their projects. The non-technologists tended to assume the design-oriented roles, and their roles in the technical implementation primarily involved programming sensor-based or musical events within the BlockyTalky environment. This allowed novices to have some control over the interaction design of the system, despite not being able to contribute to work that required more traditional text-based programming.
BlockyTalky is a powerful product that supports personally-motivated learning and a connection to the arts. BlockyTalky lowers the barrier of entry for novice computer programmers by offering high-level abstractions that prevent syntax errors, as well as the ease of plugging and playing with new sensors.
BlockyTalky: New programmable tools to enable students’ learning of networks
Annie Kelly, Lila Finch, Monica Bolles, & R. Benjamin Shapiro
In Proceedings of the 2018 International Journal of Child-Computer Interaction
BlockyTalky: A Prototyping Toolkit for Digital Musical Interfaces
Annie Kelly, Monica Bolles, & R. Benjamin Shapiro
A workshop facilitated at the 2017 International Conference on New Interfaces for Musical Expression
BlockyTalky: Tangible Distributed Computer Music for Youth
R. Benjamin Shapiro, Annie Kelly, Matthew Ahrens, et al
Selected for the 2017 Computer Music Journal
BlockyTalky: A Physical and Distributed Computer Music Toolkit for Kids
R. Benjamin Shapiro, Annie Kelly, Matthew Ahrens, & Rebecca Fiebrink
In Proceedings of the 2016 International Conference on New Interfaces for Musical Expression