Looking for similar works.

[carpal]

Hey folks, first post! I'm pretty new to the whole VJ scene, but I'm working developing an application that will match visuals to music on the fly. I'm trying to use a system of neural networks and pattern recognition algorithms to find different cross modal relationships that we wouldn't otherwise perceive.

So yeah, that's about clear as mud, right? Anyway, I'm wondering if anyone knows of other artists who work with a system for pulling up video clips in a similar automatic fashion. Thanks in advance for any suggestions or links.

Great community you've got here!

[VJbigfug]

Hi carpal,

I'm currently working on triggering video and visual effects from frequency and tempo analysis in real time.

Frequency bands are obviously the simplest to implement and result in sending out MIDI controller messages.

Tempo detection is somewhat complicated, although when achieved, and in sync, it gives a nice effect of reinforcing the energy and pace of the music. I've also been applying it to spoken word with some interesting results; working on the principle that everyone has their own range of speech rhythms. Combining overall frequency shifts with tempo/beat information should result in an educated guess being made about time signature and where the first beat of the bar is.

I've also been expanding the frequency detection to isolate and track peaks (i.e. notes) and send those out via MIDI. My next step there is to add calculation of key and chords, which can then be used to discern the role of each note within the current piece. For example playing a major third interval as opposed to a minor third. Again, all in real time.

Another aspect I haven't started on yet is tracing the frequency peaks across time to extract vibrato, tremolo and portamento, all of which illicit an emotional response depending on their timing, loudness and relation to key.

My long term aim is that by calculating data that is reasonably accurate of a given sound source, I can then use this information to move between unison (music and visuals tightly bound together) and counterpoint (music and visuals linked, but in a more interpretive manner) in real time improvisational performance.

Would be very interested to hear more about what you're up to...

Cheers,

Alex

[many2]

I'm trying to use a system of neural networks and pattern recognition algorithms to find different cross modal relationships that we wouldn't otherwise perceive.


I'd really like to hear more about your method and its advantages over other methods. Would you have to teach the neural network by feeding it some music first or will the system be closed once in the hands of users ? (I once designed an environmental themed videogame based on neural networks that never came out but since then I've always had an interest for neural networks).

[Rovastar]

I don't get it. Humans cannot decide what images/videos go best with which music how do expect to tell a computer even through AI which works best. I presume, someone (a person) has to say "yes that image works with that piece/style of music" so the program will learn. And each person opinion is highly subjective. Maybe I don't get it drop us some more info. For the record I program sound reactive computer generated visuals from sound but not trigger video.

[carpal]

I hear you guys. Sometimes I think formulating an adequate description of my project is harder than the work itself. Let me try again.

I'm not really a visual artist but rather a musician. At about the point I started getting into composing and producing my own stuff with Ableton Live, I was also thinking about the many shows I had been to that featured laptop artists and how no matter how great the music was the performance was generally not that engaging. This concerned me as it was the direction my own music was heading.

Now I've played in plenty of live bands, and know that feeling when the energy between the audience and the performer just kind of clicks. This is something I feel often gets lost when the instrument is as cold and removed as a computer. I knew that projected visuals can add a lot to the performance, but I also wanted to try and find a way to bring that link back between the audience and the performer.

Without diving into too much detail, I can tell you that I was excited about the power of neural networks when it comes to recognizing patterns and making predictions. I thought if I can employ such a technique on reading the audience (you can do a lot with the info gleaned from a live video feed), I could use the predictions of my neural net to directly influence the direction that the music is taking.

As far as triggering video clips goes, what the application is looking for has nothing to do with the subjective content of the video. What I'm attempting to do is run tempo/rhythm analysis on the live output of my music and have the software trigger video whose motion has a similar rhythm. I've sort of come up with a system for mapping the rhythm of a clip based on peaks in motion, just like one can map tempo in an audio clip with peaks in volume. Now I'm at the point where I'm trying to marry everything together. All of this is being done with Max/MSP.

So yeah, that might be more than you guys needed to hear, but that's the basic idea. Mostly I'm just hoping to find examples of other work which might run along the same lines as mine, but I'm definitely up for any other sort of discussion!

[king of sexx]

www.mbr7ob.com

[KillingFrenzy]

You know, "King of Sexxx" was my first choice before "KillingFrenzy" but I figured Americans censor sex more than violence so I'd better go the bloody direction.

back on topic.

I'm using Max/Jitter as well and getting more and more freaky with my input methods. I've been playing with image analysis, beat detections, audio frequencies and then some weird content input sources such as Google image search and speech recognition. Honestly, I think most of this makes for a better installation then a performance... but it definitely starts getting into a different headspace then the "if I push this, this will happen" way of thinking. Having your computer learn behaviours over a series of performances would be an interesting project. In other words, come up with a very "organic" system, which would have to be very complicated, with a lot of variables. Then use the following as your learning tools; audio input, crowd image, your own behaviour. So, the audio and crowd factors would be influencing the imagery, and then maybe your own decisions about the imagery (press button to make image darker, press button to make image less spikey) would start creating learned associations. By 5 performances in, your computer should be able to tell you when Xmusic and Xlighting situation occur, Master prefers Xbehaviours. Again, I see this working better in a controlled environment like an installation... but repeat performances of the same material could be seen as a controlled environment, of sorts.

[KillingFrenzy]

I can see this going very well with your work, Rova. In the sense that you could create compositions for your visualizers that would take in environmental variables (including the touchy feely ones, like what "emotion" a song provokes) and react accordingly. In a way, it is sort of an extension of what you're doing outside the "box" of audio input.