Representing music as visuals

[Dogen]

Hello all,
I’m interested in any work done in transforming music into visual sequences, or videos. An example would be a visualizer used with a music player, but I’ve found most of them crude and predictable. What I have in mind is reflecting sound visually in a way that’s subtle, responsive and very much an art form of its own. For example, pitch could be represented as higher or lower on the Y axis, harmonic content by color, and volume by brightness. Can individual instruments be discerned and represented by visual objects?

Most of us can “see” very sophisticated visuals when we listen to music. Anyone know of any interesting work in portraying and sharing those visuals on a video display?

Thanks for any pointers!
Bob

 

[pozz]

Can individual instruments be discerned and represented by visual objects?

I find visualizers are hampered by the processing time. They are usually a hair slow or worse, and have limited resolution (just like FFTs).

 

[pozz]

Jeff Minter was a pioneer. Worth checking his work.

 

[Katji]

There are people doing serious work like that, but it comes out in pro audio context, performances, no way at consumer product/ audioplayer level. ... It would need the equivalent of high-end gaming graphics processor too.

 

[ZööZ]

There are people doing serious work like that, but it comes out in pro audio context, performances, no way at consumer product/ audioplayer level. ... It would need the equivalent of high-end gaming graphics processor too.

And/Or some delay calibration function for the audio?

 

[Katji]

If it's that level/context of pro audio, then that is there already.

 

[Dogen]

There are people doing serious work like that, but it comes out in pro audio context, performances, no way at consumer product/ audioplayer level. ... It would need the equivalent of high-end gaming graphics processor too.

Who are some of these doing the work? Are there products out there that can do this, if not in real time?

 

[Katji]

Products, I don't think so. It would be whatever they find to work with.
Whenever I see some mention of it (in the context of electronic/experimental like MUTEK and so on,) and they refer to it, I look further and it usually turns out to be just the usual light show stuff - which has been done better for big events, for years. There was one I saw, not so long ago, that was actually generated images, but I have no idea what I could even search for...or I'd start with the browser history.

 

[mjwin]

Any process, which atempts to create visual stimuli solely from an audible input, will not in itself create new information, but only bombard our visual sense with derivative material. This is regardless of the complexity of the processing algorithm. Whether the addition of the visual element is beneficial to the experience is a matter of personal opinion.

Most of us can “see” very sophisticated visuals when we listen to music. Anyone know of any interesting work in portraying and sharing those visuals on a video display?

Synesthesia, the neurological phenomenon by which certain sounds can give rise to specific visual phenomena is apparently not uncommon, but is not something which I have ever had the ability to realize.

When I listen (seriously) to music or sound recordings, I close my eyes, and do indeed allow my imagination and life experience to create a unique experience. This won't be a direct recreation of the original performance, and might not in any way correspond to what the artist(s) had in mind, but that is the power of art. The experience is personal, and unique to me. Triggering the beholder's imagination is the cornerstone of art, whatever the medium.

Even given the technical ability to do so, I suspect that the recreation of my own images in someone else's visual cortex would not achieve the desired effect. It might even be perceived as visual "noise" as it would detract from that person's own imaginary response. That tends to be my own take on any attempted "sensory augmentation" of a primarily audible experience. But then I love listening, & I love imagining... (I have no TV or video screen in my home.)

Where the sound-vision process might have a place is in the performance arena itself, though even then it could be viewed as little more than an "effect". I cannot think of a good example of live performance where the visuals are derived solely from the audible. With, for example, music/dance (traditional) or music/video, the audio/visual might have some connection, but perhaps not. The two are separate entities, and work together in the beholder's imagination. A more abstract but classic example is the long-form music & light performance/installations created by Lamonte Young & Marian Zazeela in the 1960s. Their performances unfolded very slowly over time, and sound, visuals, situation (& probably imbibed substances) combined to produce the overall effect...

 

[pozz]

Products, I don't think so. It would be whatever they find to work with.
Whenever I see some mention of it (in the context of electronic/experimental like MUTEK and so on,) and they refer to it, I look further and it usually turns out to be just the usual light show stuff - which has been done better for big events, for years. There was one I saw, not so long ago, that was actually generated images, but I have no idea what I could even search for...or I'd start with the browser history.

The visuals at shows is a mix of automation and manual control. I don't know any examples where it's fully automated and algorithmic, like in visualizer plugins.