Reverse Visualisation or Sonification Leads

photonal's picture

I'm interested to doing some reverse visualisations or sonification of visual data into sound/music.

So I was wondering if anyone had some leads on some qtz compositions which would form a good basis for this. Or which technologies, e.g. OpenCV? might work the best / produce the best response in terms of low latencies and efficient CPU/GPU usage.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

photonal's picture
Re: Reverse Visualisation or Sonification Leads

(..where's the edit button? :)

For example I'd like to feed some video (could be live source too) into a composition and perhaps select sections of the viewport as data sources; multiple ones would be great. Would it be asking too much to be able to measure hue, saturation and brightness and convert to e.g. midi or OSC data?

cybero's picture
Re: Reverse Visualisation or Sonification Leads

You could look at the Image to Sound plugin from _1024 or do some vertex displacement of image to mesh.

photonal's picture
Re: Reverse Visualisation or Sonification Leads

yes thanks I looked up the _1024_Image_to_Sound QCplugin, but it appears todefault to the "first 1024 pixels of the image" ("upper left corner") so it doesn't seem to be possible to determine which 1024 pixels you'd like to analyse, i.e. make the 1024 pixel buffer 'look' at some other part of the input image source.

or as Henry Ford would put it; you can sonify any part of an image as long it's the upper left corner! ;-)

cybero's picture
Re: Reverse Visualisation or Sonification Leads

But if you placed a process before that samples distinct 1024 portions of an image, and send that to Image to Sound that would begin to fill that bill.

Doing an Image to Mesh usually involves the whole image, but thereafter the mesh is potentially accessible.

photonal's picture
Re: Reverse Visualisation or Sonification Leads

ah great idea, that's what you meant by "some vertex displacement of image to mesh" ! :)

thanks for the pointers

franz's picture
Re: Reverse Visualisation or Sonification Leads

you can actually crop your wanted portion of the image, then eventually resize it to 32 by 32 pix. (== 1024) so that it fills the audio buffer. see attached example.

PreviewAttachmentSize
32 by 32.zip17.43 KB

photonal's picture
Re: Reverse Visualisation or Sonification Leads

Sorry if I'm missing the obvious here but I don't see how I can use the analysis data from _1024_Image_to_Sound plugin as it does not seem to have published outputs, so I can't use that data to drive my sounds.

cybero's picture
Re: Reverse Visualisation or Sonification Leads

What's missing from the thumbnail sketch is a Kineme Audio Input that can pick up the generated sounds from the interpreted image and feed back the bucket aggregated lo mid hi and also send back peak, waveform and also frequency data in a variety of formats. Using Noise Industries free Random Crop allows one to have admittedly random points in the image interpreted to sound.

If you also have a built in process to audio react or affect some other part of the graphics produced prior to going again to the Image to Sound, then you have the facility to get some audio visual feedback loops .

I'm beginning to think that something along the lines of David Langford's Weighted Grid Renderer might prove useful.

And looking briefly into that, it looks like good for something else entirely :-)

franz's picture
Re: Reverse Visualisation or Sonification Leads

you can look at the dev. sample patches "color to midi" . as far as i recall, it will convert a color to midi note, or something like that. Modifying it to accept any other kind of data should be trivial.

photonal's picture
Re: Reverse Visualisation or Sonification Leads

I hacked this together as a starting basis (from a Chart comp and the 32x32 _1024 comp), but now having trouble getting midi out of the kineme midi patch - how do you set the midi port to receive the midi data?

I tried using the system 'MIDI Controllers Sender' patch but that keeps causing QC to crash.

Any clues welcome.

PreviewAttachmentSize
ChartTools-VideoHistogram.qtz17.39 KB
Waves-2.MOV896.56 KB