Week 8
Midterm project w/ Jim Schmitz
We ultimately ended up using software recommended in the book "Music and Audio Projects for Arduino" to change the voices using Jim's laptop, after realizing that sending MIDI commands to a software synthesizer like Omnisphere is a little more tricky to work with. However, we got the basic MIDI commands to work! And with the basic software it works exactly as it should. He explained the method he used to me as we went along so that I could understand it as fully as possible and also explain it when we do our demonstration in class for the presentation.
Basically, using a graphing method that can also be applied to changing stock prices or financial information, Jim calculated a "moving average" in order adjust to people's varying heartbeat graphs. The moving average takes in the newest information with a weighted average, thereby giving more importance to new data, and making it affect the average more heavily. It tracks heart rate data for a matter of a few seconds before throwing away old data, giving the newest data the most weight (0.9, for example) and the oldest heart rate data the least amount of significance (0.1). This allows the Arduino to detect that a heart beat is about to spike when it starts to move a certain amount above its average, and then send a MIDI signal to the software synthesizer to play a note each time the heart beat spike rise is detected. The code also tells the note to stop playing a certain amount of time after it starts. When the moving average starts to go downwards, the code sends a MIDI command telling the software synthesizer to reset and start again to wait for a new heartbeat. The code also tells it through a MIDI command to change the key within a certain range (50-62), and also change the voice (within certain voices we chose out of the bank provided by the software synthesizer that works well with Hairless MIDI Serial) and also add sustain (a basic MIDI command). The way MIDI works is that the first byte sends the command, and the second byte tells it what to do. So, for example, if the byte starts with 1, it says to do *something*, such as play a note. The next byte says what note to play, and the volume. The Voice button, however, only sends one command - to change the voice (until it exceeds the "voice list" length, and which point the code tells it to return to the beginning of the list of sounds).
The math for the heartbeat detection study we did for this project is thoroughly documented on Jim's blog here.
Here are some audio samples from both us experimenting with meditating using the Midi Meditation (using two different sounds in Omnisphere):
And some videos of us interacting with our prototype...
DATA (USER TESTING)
1) Below are graphs showing our first user testing graphs (without the moving average applied):
2) User testing graphs with the moving average applied (to show how the code will detect the varying heartbeats using the moving average):
I think what really brought this project together was that we came up with an interaction that we were actually excited to work with and use ourselves. It also helped (a lot!) that Jim had a background in working with moving averages, which allowed the project to work more how we imagined and essentially allowed the interaction to come to life. Our original idea of having the heart rate trigger the tempo of a piece of music or notes within a scale to create music changed during brainstorming for the project to the idea of triggering a single sound in time along with a person's changing heart rate, and that seemed to make the interaction more interesting to us and for others as well...I think our only hitch was working with the Omnisphere software that made it more difficult and slowed us down somewhat, but as it ended up when using the other software it worked just as we intended. I also learned some code, and about how MIDI works. I actually think I now understand how each MIDI command worked and how we programmed the buttons, and also some things clicked when wiring the bread board and soldering for the first time.
We realized also that it was actually helpful to have the heart rate sonified this way, or calming, as this interaction and sound in effect reminds you to be more aware of your actual heart beating. And in this version it is also more conducive to using for meditation or experiments with meditation.