Today's modern technologies are internet-connected and embedded with various sensors that gather data about our environments and bodies. Despite this, computer science education hasn't changed to address these new and exciting technologies. Most people do not learn about physical computing sensors and networking until advanced courses in college.
Educate and empower youth to learn about IoT and sensing technologies, and to build their own custom systems that leverage these technologies and are personally expressive and meaningful.
The first designs of BlockyTalky built on computer science education research that says block-based programming abstractions lower the barrier of entry to new programmers. We also built on connected learning and constructivist principles that argue students learn more effectively when they can connect their education to personally meaningful things in their lives.
The BlockyTalky programming language is easy to learn. We designed it to foster computational thinking and problem solving by offering a tight match between the way novice programmers describe the interactive systems that they want to make and how they can express those ideas through code. And our graphical programming language helps students to avoid syntax errors.
BlockyTalky enables users to simply plug-and-play with a variety of GrovePi sensors.
In 2016, I worked with a middle school music teacher to co-design a set of new features for BlockyTalky that would better suit his music theory pedagogy in the classroom.
When I became the lead designer and developer of the BlockyTalky project, I began to investigate how we could rethink the design to make it support adult artists and musicians who wanted to
build interactive creative systems.
To investigate this design space I took a mixed-method approach of naturalistic observation, surveys, focus groups, and one-on-one interviews. During this phase of research I identiified some ways new media artists were using interactive systems, such as visual jockeying (VJing), gestural music control, and wearable technology.
The problem was that developing these systems can be expensive and incredibly complicated to the novice programmer. I decided to design high-level abstractions that would let artists send data from BlockyTalky sensors to outside creative software using Open Sound Control (OSC). This enabled BlockyTalky to connect to programs like Max/MSP, Unity, Processing, and more.
I tested my new BlockyTalky designs by hosting a creative hackathon at the University of Colorado Boulder called Creative++. Monica Bolles and I designed the hackathon to be beginner-friendly and advertised the event to artists and musicians, as opposed to programmers and engineers. We introduced participants to the BlockyTalky environment, facilitated a project brainstorm session, led a group formation activity, and then gave groups twelve hours to build a project that used physical controls and sensors to create some type of musical or artistic output.
We collected participants’ pre-screening information to provide us context for the demograph- ics of participants including their age, gender identity, college major or career and responses for their answers regarding why they want to participate and what they hope to learn. In addition, we conducted semi-structured interviews of participants, collected questionnaire responses, collected project documentation such as photos, code, and notes, and captured recordings of groups’ final presentations and performances. We gave participants the questionnaire at the end of the event, therefore a few people who left early did not have the opportunity to fill one out. The questionnaire asked identity- and project- specific questions like, “have you ever programmed something before this event?” “do you identify as a programmer?” “have you made art before?” “do you identify as an artist?” “what did your group make?” “what was your role on this project?” “what were your teammates’ roles?” and other questions asking about their opinions on the usability and experience of using BlockyTalky.
The addition of OSC to BlockyTalky enabled groups to create interfaces to control profes- sional artistic software like Max, Processing, and Unity instead of having to rely on the musical capabilities of the BlockyTalkys themselves. This ability to interoperate with more sophisticated software enabled the rapid construction and customization of live controllers for expression, demon- strating its potential beyond K-12 settings for use by adult creators. As expected, the projects that participants created still required at least one technical expert on the team. However, all participants were able to contribute to some level of the technical implementations of their projects. The non-technologists tended to assume the design-oriented roles, and their roles in the technical implementation primarily involved programming sensor-based or musical events within the BlockyTalky environment. This allowed novices to have some control over the interaction design of the system, despite not being able to contribute to work that required more traditional text-based programming.
BlockyTalky: New programmable tools to enable students’ learning of networks
Annie Kelly, Lila Finch, Monica Bolles, & R. Benjamin Shapiro
In Proceedings of the 2018 International Journal of Child-Computer Interaction
BlockyTalky: A Prototyping Toolkit for Digital Musical Interfaces
Annie Kelly, Monica Bolles, & R. Benjamin Shapiro
A workshop facilitated at the 2017 International Conference on New Interfaces for Musical Expression
BlockyTalky: Tangible Distributed Computer Music for Youth
R. Benjamin Shapiro, Annie Kelly, Matthew Ahrens, et al
Selected for the 2017 Computer Music Journal
BlockyTalky: A Physical and Distributed Computer Music Toolkit for Kids
R. Benjamin Shapiro, Annie Kelly, Matthew Ahrens, & Rebecca Fiebrink
In Proceedings of the 2016 International Conference on New Interfaces for Musical Expression