_1024_Image_to_Sound plugin

franz's picture

Hey, still highly experimental, this plugin will convert an image buffer into an audio buffer. Right now, only the first 1024 pixels are interpreted as sound... :)

get it here: 1024 blog

gtoledo3's picture
Re: _1024_Image_to_Sound plugin

Thanks! Looking now...

gtoledo3's picture
Re: _1024_Image_to_Sound plugin

Aww, it doesn't restart after the Viewer has been shut off.

I notice that Viewer initialization stuff seems to be an issue with the sound plugs... I think your tone generator did this, and the OFX thing was a little different in that I think it wouldn't shut off until you quit QC or something?

What exactly is it doing? It sounds like it produces only one pitch, and is filtering the waveform differently depending on the image? It sounds like when you push a digital resonance filter into aliasing sort of.

Thanks again!

franz's picture
Re: _1024_Image_to_Sound plugin

hey gt, thanks for testing this ! yeah, the audio part is the same from ToneGenerator plugin. It actually uses AudioToolkit.framework to render audio on its own thread. That's why it might be picky when stopping viewer. I'll check that, thanks for noting.

The plug is actually reading the image pixels starting from upper left corner and putting the luma value (r+b+g / 3) in an audio buffer (size =1024). So the pitch will depend on the amout of white pixels in the image. ( a large horizontal bar will produce a low pitch )

cybero's picture
Re: _1024_Image_to_Sound plugin

Interesting plugin. Like the way you can get different audio results by changing the LFO settings.

Does the image have to be black and white only?

[postscript - posted example shows how colour can be brought in and change sound produced, so question answered]

How can we ensure that the tone plays if we change state of the LFO or the image feed?

[postscript - posted example shows how image feed can be changed and change sound produced, so question answered apart from any LFO related stuff]

gtoledo3's picture
Re: _1024_Image_to_Sound plugin

No problem, my pleasure!

...that's why I was asking about the pitch thing. It really sounds like it just puts out one pitch, and that the image actually changes the resonant peak, not the resonant pitch. I admit, I'm still in "wake up/drink coffee mode", so I may be mis-hearing/mis-testing, but that's my initial take.

This is a great direction to go in!

I may as well mention it on-forum, b/c you seem so into the audio plugins lately- echo, delay, phase, flange, and chorus are all basically possible right now, inelegantly. Someone should go ahead and write something that takes a track, creates a double that plays along with the original track, and allows floating point level offset of time of the double, as well as detune (ala the stock Apple audio pitch control that's in SL), it would allow for a range of time based effects.

Outside of the scope of what's really possible now, if audio was a provider to a render patch, and a supported data type, we could be making things like Lexicon multi-tap delays emulations in QC by using the Queue, etc... This is something I would really like to see fleshed out.

cybero's picture
Re: _1024_Image_to_Sound plugin

Just discovered that using the textured image as a mask works really nicely.

What really adds a different twist is using the v002 Glitch Core Video OpenGL at 1920*1280 width to provide the main image and use the textured image as the mask, that actually affects the sort of sound produced.

Definitely see a working place for this and the use of Glitch as main image seems to overcome some of the other problems referred to previously about having to reset the player to play sound.

I had to keep on closing and re-opening QC to get the example to work again.

Find attached my redraft of your example, franz. Uses an Image Multiplex to cycle through various working forms of v002 Glitch

Hope it proves useful, demonstrative to someone or another.

franz's picture
Re: _1024_Image_to_Sound plugin

any links on the phaser/flange ? do you mean doing it with QC patches, or handcoding ?

Having said that, I wish KnM to release their latest skanky SDK, so my plugs. could be compatible with KnM custom datatypes. Not likely to happen though...

EDIT: btw, I just thought that the opposite could be done: audio mic input to image.... mmmm, I`ll investigate.

EDIT2: coffee ? aren't you in europe (it's beer o'clock atm in GMT+1) ?

franz's picture
Re: _1024_Image_to_Sound plugin + vade's GLItCH

nice, here's a modded version of yours, using vade'GLITCH. Just added the glitch to the output. Press space to play sound+picture glitch.

as a sidenote, i'll check further on this viewer ON/OFF problem, cause it is really bugging me.

cwright's picture
Re: _1024_Image_to_Sound plugin

franz wrote:
...the luma value (r+b+g / 3)...

pedantic : that's not the equation for luma, that's an average (luma is 0.299 * r + 0.587 * b + 0.114 * g for ITU601, and something close but slightly different for ITU709)

These all have slightly different results (and one isn't clearly better than the other), but it's definitely not luma.

PreviewAttachmentSize
Average vs Luma.png
Average vs Luma.png155.28 KB

cwright's picture
Re: _1024_Image_to_Sound plugin

franz wrote:
Having said that, I wish KnM to release their latest skanky SDK, so my plugs. could be compatible with KnM custom datatypes. Not likely to happen though...

Not sure how it is now, but I know that the plugin data types were never a part of the sdk when I was working on plugins -- they were always built into the plugins themselves (the only exception was particletools + kineme3d, and that wasn't even sdk-level, but a hack between those two plugins specifically).

franz's picture
Re: _1024_Image_to_Sound plugin

thxx for the precision. I'll stop begging then. (even if a link on how to do my own datatypes, would be, errr ... tremendous !, but maybe that's not the current aapl policy)

EDIT: found a link for digital flange : http://www.donreiman.com/Flanger/Flanger.htm

franz's picture
Re: _1024_Image_to_Sound plugin

that's why I love you. Spread the knowledge.

Is luma on the left in your image ?

gtoledo3's picture
Re: _1024_Image_to_Sound plugin

To clarify, I meant doing this manually by offsetting time and pitch with stock patches, kineme audio/stock audio/stock video in asynch, all to varying levels. Time based audio effects can be done in a clunky and inelegant way.

The other aspect though, is that for some reason the built in audio patch sounds way crappier than I remember it lately - it always sounds like it's clipping! So, the pitch tuning is pretty of limited use, since the audio playing out sounds crappy. All audio patches available also seem to create a kind of audio glitch when you try to trigger it on via a key click or something, intermittently, except maybe the video patch in asynch.

Nope, in FL. ;-)

cybero's picture
Re: _1024_Image_to_Sound plugin + vade's GLItCH

Cheers - a quick screen grab of my redraft, just about to have a look at yours.

cwright's picture
Re: _1024_Image_to_Sound plugin

Yeah, unfortunately I can't talk about the details of doing it (though smokris knows more than enough to do a writeup, if he has time).

There would of course be other things to learn from the SDK, but using existing kineme ports isn't one of them :/

cwright's picture
Re: _1024_Image_to_Sound plugin

Luma's actually on the right in that picture (with average on the left) -- if I changed the image loader to not do color correction, they both looked quite different.

Like I said, there isn't a "best" for this sort of thing, it just depends on the look you're going for. :)

franz's picture
Re: _1024_Image_to_Sound plugin

unfortunately, smokris doesn't seem to be in the QC business anymore (at least not as much as you both used to be). Maybe that's one more reason for me to come visit you american folks one of these days. ;)

gtoledo3's picture
Re: _1024_Image_to_Sound plugin

WWQC!

franz's picture
Re: _1024_Image_to_Sound plugin

just uploaded a new version that takes either average pixel value, or RGB consecutively. Since buffer size is 1024, the incoming image should be 32*32 pixels.

dust's picture
Re: _1024_Image_to_Sound plugin + vade's GLItCH

very interesting plug_in, will have to investigate further. cybero your getting some interesting noises in this video. haven't tried this yet but might be interesting to try an isolate some particular images to syncopate some noisy rhythms with this.

cybero's picture
Re: _1024_Image_to_Sound plugin + vade's GLItCH

It's got shed loads of possibilities, colour , blending, layer order, all can make a difference to the sound produced from the image feed processed by the Image to Sound plugin.

Then there's the type of image, static, moving, generated, all makes a difference to the sound actually produced.

Add in an audio input patch and feed the tone back into the image creation routine whose product is being fed into the Image to Sound plugin.

Hypno Toad :-)

dust's picture
Re: _1024_Image_to_Sound plugin + vade's GLItCH

yeah i was actually doing the same earlier as per the feedback loop generating the images generating the sound generating the images.... i did some experiments a few years ago with with max msp. taking the rgb average mean from a luminance source like a light bulb to generate sound with a video camera based on those values. it was interesting, i was able to use a potentiometer type dimmer light fader to ramp the sound up and down with a light switch sort of like a theremin. very hard to play classically.

after a few years experimenting i finally figured a motion detection kind of thing like optical flow tracking a grid that has particular keys in a one to one sound or midi map has proven to be the most harmonically sound and easy to play traditionally but this plugin is something totally different. without knowing or seeing the source code its a bit hard to actually determine the cause and effect response to images.

it seems color only has an effect for me on the overall amplitude of the sound, where as size and rotation seem to have direct cause on the pitch. although the image is resized before it hits the plugin it seems only the top few rows of the pixel data is used to make the sound. at least as far as i have seen by messing the dev comp.

either way defiantly interesting. i bet smorkis has been having all kinds of fun with this plugin.

franz's picture
Re: _1024_Image_to_Sound plugin + vade's GLItCH

code works like this: for each pixel, it will copy the float value to the audio buffer, till the buffer is filled (so it takes "only" 1024 samples, because the audio buffer is 1024 samples large). that's it.

maybe this example is better (see attached file)

PreviewAttachmentSize
32 by 32.zip17.43 KB

cybero's picture
Re: _1024_Image_to_Sound plugin

Here's an image to mesh | image to sound | histogram to tones setup. Needs the Image to Sound & the Tone Player plugins & 10.6.x [Meshes].

Uses the Text Extrusion Vertex Displacement kernel from Apple.

PreviewAttachmentSize
soundmoviebank.qtz74.61 KB

franz's picture
Re: _1024_Image_to_Sound plugin

I just fixed the restart problem. Now everything should be running OK.

cybero's picture
Re: _1024_Image_to_Sound plugin

will give it a try out as soon as I've finished the last of my offline renders for the day.