Author: Elora Trotter

Mosaic is an application designed by Emmanuel Mazza that combines Live Coding with Visual Programming in order to streamline the creative process without sacrificing artistic vision. Based on OpenFrameworks, an open source creative coding toolkit founded by Zachary Lieberman, Theo Watson and Arturo Castro, it utilizes a wide variety of programming languages including C++, GLSL, Python, Lua, and Bash. As seen in my patch above, it works by connecting blocks of sonic and visual data, facilitating an easy-to-follow data flow. Because Mosaic allows for more complex possibilities while remaining accessible, it is suitable for both beginners and advanced users who wish to create a wide range of real-time audio-visual compositions.

The diagram above outlines how Mosaic hybridizes Live Coding and Visual Programming paradigms in order to reinforce feedback and promote human-machine interaction. Its page, linked here, quotes Henri Bergson: “The eyes see only what the mind is prepared to comprehend.” For someone like me, whose comprehension of programming is limited, I am grateful for applications like Mosaic that allow me to create projects I can understand.

Having previously worked with MaxMSP, I found Mosaic’s interface buggy but still intuitive and to use. For my project, I wanted to create audio-reactive visuals that convey feelings of nostalgia and loss of memory. I found this old video of my father recording a song he had made for my brother and me. In Mosaic, I navigated to ‘Objects’ and then ‘Texture’ to find all the nodes that could manipulate and export video.

As seen above, I juggled various concepts and imported multiple videos to explore how Mosaic is able to warp and blend textures to serve whatever concept I landed on. I really liked how the video grabber blended my real-time position via the MacBook camera with the singing video of my father to convey how memories stay with us even as we change and grow. Because Mosaic can only play sound and video separately, I extracted the audio file from the video using VLC media player, and started focusing on how I wanted to manipulate the audio to convey a sense of loss.

As seen above, I used the compressor and bit cruncher objects to add distortion to the sound file so that I could lower or amplify the distortion real-time by lowering the thresh and moving the slider. The entire time, I was reflecting on how if I was using with a platform that only allowed for written code, like TidalCycles, I would have to manually write out these effects, but using Mosaic, I could drag and drop the objects that I wanted and simply connect them to achieve the control the audio the way I wanted to.

The most difficult part of my project was figuring out how to connect visual components with audio so that I could manipulate the blended video of myself and my father as I increased or decreased distortion. I really liked this audio analyzer object because, as seen by the yellow input and green output, it allowed me to do just that, and as an additional bonus, it manipulated the video by sound that were playing real-time, so I could speak into the Mac microphone and the video would distort even further. I really liked how this object strengthened the concept of my project, because I could speak about memory loss and the video would distort even further in response.

The audio analyzer object converted the sound data into numerical data that could then be converted back into visual data. And I blended visual distortion with the video of my father by controlling the sliders as seen above. I really loved how accessible the controls were, allowing me to manipulate the video and sound real-time according to the demands of the performance.

The finalized video and audio components of the project can be viewed here and here respectively. Through live manipulating the video and audio with my voice and the Mosaic controls as seen in the footage, I was able to convey concepts like memory loss and nostalgia for the purposes of this project. I really loved the potential for creativity via Visual Programming that Mosaic offers, and I will definitely continue to use this application for personal projects into the future.

When I was in middle school, during one of his rare visits, my father showed me an Aphex Twin song from Syro. At that point, having grown accustomed to Ariana Grande and Justin Bieber radio hits, I had said, “It’s just noise.” He responded, “You’ll be able to see the patterns–the music–someday.” My brother and I would sit in front of our battered Bluetooth speaker and listen to Aphex Twin songs in order to understand what our father possibly saw in these strange metal-like songs. We would point out a sound when it arrived what at first seemed too early or late. We would gape at sounds that surprised us because they arrived and repeated in ways we hadn’t expected them to, and gradually, we began to love this sense of musically organized disintegration. Ariana Grande and Justin Bieber no longer––that bored us. My brother eventually became a jazz drummer and obsessed over the likes of Miles Davis and John Coltrane and is now majoring in music. I trace his interest back to this anecdote.

Aphex Twin is inseparable from experimental electronic music, and yet, despite being so electronic, his work is undeniably intimate and human. I believe one of the reasons for this is his genius knack for variation and timing, something he learned, as the reading pointed out, from African rhythms. Stockhausen wasn’t the biggest fan of Aphex Twin’s work because he claimed he should “stop with all these post-African repetitions” and should “look for changing tempi and changing rhythms.” But Aphex Twin’s ability to continuously repeat interlocking rhythms with genius variations is why people have come to love and idolize him so much. Not everyone can do so much with so little. Another artist that this reading reminded me of was Tune-Yards, who also draws from African rhythms to create human-like complexities and variations in her work. She has a lyric that goes: “I use my white woman’s voice to tell stories of African men…” As we go full throttle into a tactile-less, screen, technocratic world, I believe prioritizing the HUMAN in music is more important than ever. Or, not even important, but the kind of music we will increasingly seek out because we need it. I was never a fan of dubstep and EDM because they were too saccharine and simple. I want imperfection. I want the human body, with all its limitations and attempted breaking of those limitations. I really enjoyed this reading and the insight I was able to glean into the music I love through it.

I spent a lot my time in New York City hanging around Washington Square Park between study breaks from Bobst. One of my most memorable encounters was with a jean skirt-wearing Jewish fella who liked to dance and spoke about how they believed their dead grandmother still lived in their hands. I think they danced because of this idea––that they were, in a way, facilitating communication between their grandmother and the audience through their living body. I commented on how this form of person-to-person interaction was more important than ever in our time of image saturation. Today, between the Internet and social media, we are constantly inundated by what I call “dead” material, or, material that has left the alive, breathing body into fixed positions, such as poetry that has been written down, or photographs. While these mediums are beautiful and important, the links between living things to other living things have been increasingly replaced with various methods of digital pseudo-connection, which could help explain the loneliness epidemic. The true poem is the one that Walt Whitman was constantly rewriting and sharing from his heart. I believe in tangible communication. I believe in dance.

That is why I believe in live coding. I signed up for this class in order to use computers towards this end. I really resonated with the idea that “we do not use computers; they use us…” All we have to do is look at how dependent we are on our phones to realize that we might be the ones being used. But computer manipulation is a way of responding back; of admitting awareness in this current technological landscape, and saying it takes two to tango––I am going to shape you just as you shape me. Within the confines of capitalism and production, most technology is used to manipulate and exploit, but within the framework of this class, I am really excited to collaborate with technology and engage in it as a means of physical, living communication. Get out of the DMs and into the techno raves! I say. One last thing I wanted to mention was the notion of “Being a User,” which entails acknowledging that there “there is, whether visible or not, a computer, a programmed system that you use.” All of us live within inherited systems and ideologies that perpetuate a lot of destruction and suffering. Whether we realize it or not, these ideologies form our thoughts, dreams, jokes, and very realities, and it is our responsibility as thinking, alive humans to question and challenge these often-invisible frameworks so that the world can actually start to change for the better.

To be a User is to be alive. Becoming a live coder means becoming an active participant in and hopefully challenger against whatever systems you find yourself in. It means observing with intent and responding real-time. This reading inspired me to keep this philosophical groundwork in the back of my head as we all start to exercise this practice.