Day 377 @ ITP: The Code of Music

Assignment #2: Audio-Visual Instrument


For this assignment I decided to start with a simple musical “stamping” tool that would change colors and notes with mousePressed. I had the initial thought to make music with particle systems that you could “paint” with or stamp across the screen. I initially had some difficulty getting the first particle system I chose to work with to change colors with keyPressed, instead it would change the colors of all the particle systems that had already been placed. Eventually I found another code that lo and behold worked when I pieced together with the rest of my sketch…still getting the hang of how particle systems work. Below is a visual timeline of my process or code “sketches” for the final sketch:

Video demo of the final version:

Here is the link again to the sketch, where you can also view the code.


I got pretty much where I was trying to go with this except the sound interaction isn’t quite where I would like to be. I think the overall palette could still be improved and I would like to work on it more in another iteration, possibly using other effects in tone.js in particular to make the single tones repeat or pitch up/down in different ways instead of rerecording new samples.

I also encountered an issue where sometimes I would hear a glitch in the audio especially while playing different sounds at once and am wondering how and if I could prevent that.

I think however this was fun to work on and could be a good jumping off point for other projects. I really enjoyed the examples we played with in class ( and floradrift), where you can play an instrument and make parts turn on and off and light up different colors, and I would like to try something in that style as well, or possibly a combination of tools with a selection menu to choose your tool (See: KidPix.). Triggering different shapes and audio-reactive visual responses from the shapes would also be nice to experiment with.