My final assignment for this class is an audio/speech reactive visualiser using p5.js, using typography and variable fonts to highlight dynamic melody lines and volume dynamics. I would imagine a singer might use this tool while performing in order to create real-time dynamic captioning.
I used the p5.Speech library to process the words of the audio while also using the p5 Sound library to process the fundamental pitch and amplitude of the microphone signal. A gate is used to determine note lengths and after collecting the levels and pitches over a set amount of time I was able to use each letter in the detected word to resemble the pitch and volume journey through the word/phrase.
Words that the speech detection algorithm is not sure of are rendered lightly with low opacity. The pitch is portrayed by the lowercase heights of each letter, and the range in volume is portrayed in the font weight of each letter.
Obviously the speech detection is not perfect and is not directly tied to the gate code, and there are a number of other things to add (see below) that would polish off this sketch but I think it’s a good proof of concept.
Base: