|
|
Low / high pass filters Hello all, For a composition I'm doing, I need something akin to a clamp or high pass filter. What I mean is a patch that'll accept an image and a color input, and then output a pure black / pure white image using the color as a threshold. For my application, this would be used with an image that's already b/w, so it could use e.g. only the red channel. I've tried my hand at a Core Image filter that does this, but I'm utterly bewildered since the documentation is so scant. I keep getting errors about data-dependent conditionals and swizzles. Is it really impossible to return a value from a CI filter based on a conditional statement? Or is there an obvious way of doing this I'm missing? In other software, I'd use a curve adjustment or similar, but I haven't found an equivalent in QC. The color controls is too fidgety and dependent on the input image. Any pointers would be appreciated.
|
Cwright posted this comp recently on the Apple QC list so I'm assuming it's legit to post it here. It demonstrates use of conditionals inside a CI filter. I agree, the Apple documentation is scant/quite abstract in nature for non comp-sci types like myself. More examples would be good, I tend to learn from pulling examples apart as much as those kind of docs.
You just need something like this really:
kernel vec4 threshold(sampler image, float threshold) { // sample input image at working pixel vec4 px = sample(image, samplerCoord(image)); // get pixel luminosity float lum = dot(px, vec4(0.299,0.587,0.114,0.0)); // Return black or white return (lum > threshold) ? vec4(1.0) : vec4(0.0,0.0,0.0,1.0); }threshold will be in the range 0.0 to 1.0.
The key is the use of the ternary (?) operator instead of the more traditional if( ) { } conditional syntax.
a|x
Nice example toneburst, I find that I have to set the LFO just above 1 [range 0 to 1.1 or offset 0.1] to obtain a longer term black on the pass. Interesting tip about the ternary operator.
Regarding tutorials for CI, sbn, http://cycling74.com/2007/05/23/your-first-shader/ is as good a place to start as any, although actual code snippets and examples are very useful, nonetheless.
Thanks for the link cybero- I'll check that out.
Incidentally, the code above will produce aliasing, since it's binary (just black or white output). You could use smoothstep() to smooooth things out a bit.
kernel vec4 threshold(sampler image, float threshold, float smoothness) { // sample input image at working pixel vec4 px = sample(image, samplerCoord(image)); // get pixel luminosity float lum = dot(px, vec4(0.299,0.587,0.114,0.0)); // return black, white and gradated grey in between return smoothstep(threshold, threshold + smoothness, lum); }smoothness can now be used to smooth out the edges of the result.
a|x
Oh, just realised- I've looked at that Cycling74 page before.
That's GLSL, so won't quite work as-is in a Core Image Kernel/Filter patch. Althought GLSL and Core Image Filter Language are similar, only GLSL Fragment shaders are roughly equivalent to Core Image Filters, so you might find a lot of tutorial material on GLSL covers a lot of stuff CIFilters can't do, and therefore isn't strictly applicable to CI.
That said, there are a some simple GLSL fragment shaders out there that can relatively easily be converted to Core Image filters.
In terms of actual CIFilter documentation, I never really managed to find much, even on the Apple site. There are certain aspects of CIFilter operation I've never fully got my head around.
a|x
Thank you all for the replies.
It turned out to be a bit of bad timing on my end, I've been putting out fires at work and had to put this on hold for a bit.
I had this idea for a difference matte implementation, but just ran in to one of QC's blind spots as often happens. Really, low / high / band filters would be most useful.
Especially the examples are extremely useful. Much obliged. I can see I need to read up on the ternary op, as well as look at the tuts.
All in all, thank you.
No prob.
It took me a while to get my head around ternary conditionals. I find myself using them in JavaScript, now too, though, since they're neat and compact.
a|x
Had a little play with this one... think you might like ;)
That's nice- with antialiasing. Cool. I can see potential there. I might do a multi-pass version of that, I think.
a|x
Incidentally, it's funny you should talk about 'high/low-pass filters' in this context. I'm guessing, like me, you come from an audio synthesis background. High and low-pass filters mean something quite different in image-processing terms, but I knew exactly what you meant because I've used subtractive synthesis methods in audio a lot over the years.
It's funny how terminology crosses-over between disciplines (and sometimes doesn't).
a|x
perhaps I'm rusty, but high-pass/low-pass (and band-pass) means the same thing in audio that it does in image processing -- audio is 1D while images are 2D, but the principles of filtering are the same.
In the OP's post, it was more of a thresholding problem (nothing to do with actual band filtering).
low-pass-filtering an image means removing the high frequency components (the "detail" so the image looks blurred sort of), while high-pass-filtering removes the low frequency components (the slowly-changing portions, so the image looks like an edge-detected variant of sorts). - http://local.wasp.uwa.edu.au/~pbourke/miscellaneous/imagefilter/ (pbourke, as always, is prolific ;) -- neither of these are taking place in this context.
Perhaps audio synthesis is different, but if so, I'm inclined to say they're "wrong", as high/low pass filtering hasn't meant anything other than the above for a century or two for both 1- and 2-D data sets (and more)
All that is true cwright, but it's the way the terms are being used here implies an equivalence between luminosity (in graphic terms) and frequency-content (in audio terms), whereas, as you rightly point out, in strict image-processing terms, hi and low-pass filtering apply to the low and high-detail areas of the image, rather than areas of differing intensity.
I'm hazy on the details of all this (and much less than hazy on the mathematics behind it), but I do like the idea that if you analyse images in the frequency domain, it's possible to apply a lot of the same processes used in digital audio manipulation. Actually, some way of converting audio>video>audio in QC would be really cool, thinking about it.
a|x
(note that I'm not trying to be a hard-nose or anything)
For operations such as this, where no frequency-domain (or even time-domain) transformation takes place, using the term "high pass" or "low pass" is misleading at best -- each pixel is evaluated entirely independently of the others. A more fitting term would be thresholding.
Converting audio to/from images is something I need to talk smokris out of every few months. I do this because, while it sounds incomprehensibly awesome and amazing, it's actually quite lame; let me 'splain.
Every conceivable operation that you do on an image can already be done on audio. Every image from audio would be 1 pixel tall (or 1 pixel wide), which makes most filters stupid. If you make non-1D images, you'll get periodic effects that may be interesting, but are still better served by actual audio processing.
There's also the overhead of transferring a chunk of data to/from GPU. This overhead is awful for audio due to the latency involved.
The only really interesting aspect of this is the idea that you could roll your own filter, and have it be a makeshift fancy audio filter. I contend that a much better solution to this would be an "audio shader" type function, where you can do per-sample programmable operations. Audio Units kinda do this (as do VST plugins), but having a runtime language rather than a compile-time language would make it much more interesting.
Didn't think you were, don't worry.
Absolutely. Exactly the point I was making, in fact.
Not quite. If you do an FFT on the audio, you can represent each frequency-band of the audio signal as a 1D row (or column) of a 2D image. There are quite a few audio applications that allow audio>'frequency print'>audio resynthesis, with various image-based editing/mangling tools that can be applied to the intermediate stage. Years ago, there was an application called Metasynth that was based around this idea, but it's also been used in audio-restoration software for a while, too.
That's the kind of thing I had in mind, in my simple-minded way.
That's a nice idea. I'm thinking another C-like language, maybe called something like AUSL.
a|x
When you generate a 2D FFT from a 1D audio source, you're "cheating" (i.e. you need to do windowing) -- windowing requires working with groups of samples, so there's bleed-over between neighbouring rows. There's nothing wrong with this, mind you, and it's used quite successfully as you've noted. However, it's not a straightforward lossless operation, iirc (it's been half a decade since I've studied/used digital signal processing though, so I'm definitely out of practice on this)
If you just do a naive audio-samples-to-luma image generation, operating on the resulting image doesn't serve much of a point. Doing a frequency-domain image would definitely be useful though (in fact, I don't know why I've overlooked this aspect, as it's quite obvious and frequently done...) Thanks for noting it :)
Heh, what am I thinking.
Of course you're right about the terminology. I even knew this had I just taken the time to think it over.
Yes, I've dabbled in audio before (Reaktor, Max, Buzz), but it's been a while. Maybe the audio way of thinking is more ingrained than I thought. These days I work in images exclusively.
ETA: That goes for both of you, thanks for correcting the terms. And for he interesting discussion.
Yes, what I was asking for was simple thresholding. In a broader sense, what would one call this sort of operation, if we included the more advanced "levels" and "curve" operations? Value-mapping?
If it were extended so that it would pass values above/below the threshold unchanged. Value clamp?
If it had two thresholds and would pass whatever was between the thresholds. Dual clamp?
It's this sort of thinking that got me thinking about filters. And while I accept that it would only confuse matters to call it that since there's already a use for that word in imaging, in any other sort of programming I'd call that sort of operations "filtering" e.g. a list of numbers even though no frequencies are involved. Hmm - maybe I should try to change my thinking and call that "mapping" or "clamping" instead.
if you're curving it, but there's a 1:1 correlation, it's mapping (there's possibly another term for this, but I don't know it off the top of my head).
with selective passing (of low values, or high values, or values between two thresholds, or values outside of two values), it's called multi-level thresholding (and sometimes multiband thresholding, if you're working with multiple bands of data -- Red, Green, Blue, Alpha would be 4 "bands", for example).
in photography, "filtering" is generally used to affect mapping, but also does some non-per-pixel operations (diffusion and cross-screens, for example, do fancier effects), but they're still not filtering in the signal processing sense (i.e. there's no "low pass" filter in photography that I'm aware of, though it'd be pretty awesome :))
(disclaimer: I was a photo nerd a bit in high school)
Good to know.
In the case of photographic filters (at the point of recording), I suppose you are filtering frequencies in a very real sense, attenuating some frequencies of light more than others.
BTW, since we're on the topic: What's your guesstimate on what's more efficient: Using the stock CI patch, or wrapping a CI object in a custom patch?
I ask because having basic threshold / mapping nodes would be neat. I'm unsure if it's more QC-like making separate patches (in which case a few macros would suffice), or a monolithic patch with modes for different value transformations. What do you think?
regarding filter -- I suppose so, though no one thinks of light frequencies unless they're really into physics (high pass = UV passes, low-pass = IR passes) ;) so it's still a per-pixel type operation, not a "remove low frequencies from this image" operation. (semantics, I know, I know -- I'm not overly attached to either, just voicing how it "typically" is)
regarding optimal performance: do both, profile.
nice circle that would make a good high/low pass filter knob for an audio filter.
im taking graduate class in perceptions this semester and well i have only been in the class a week but it seems not only can image and audio filtering be applied to each other but psychophysics also uses these terms which seem to be interchangeable as well. psychophysics is basically the study of how external stimuli can effect us mentally. yeah it sounds a bit weird but thresholding, signal detection both indiscriminate, and discriminant, as well as multi-dimesnsional sampling techniques are used etc.. at least the principles of psychophysics hold true to both sound and vision if you use Fechners law. Sure the math is the same both in the audio and visual world but it ultimately comes down to how these audio visual filters effect us. Im still a little unclear of how we are going to measuring things like this but i defiantly know how a lowpass resonant audio filter can effect my stimuli internally.
Ha! I know what you mean. I'm a bit of a synth nerd, and those simple subtractive synth sounds get me every time.
a|x
Ah, OK. I've heard the term, but was never sure exactly what i meant.
You'd essentially be doing AM (amplitude modulation) on the audio, I imagine. And probably massively downsampling it too, unless you were working with very long 1D images (that were different lengths per-frame, depending on framerate).
That was what I had in mind. Should have made this clearer from the beginning. In fact, you could make some kind of 16-channel/band 'spectral-print' visualiser with QC's builtin tools, I think. The trick would be to resynthesise than back into audio after it had been modified in various ways.
Incidentally, I don't know if you've ever played with NI Reaktor, but it has the facility to code custom DSP modules. I've not investigated it, but I have a feeling it might be a proprietary version of something like the 'audio shader language' we were talking about. Just thought of that.
a|x
unfortunately, there's not enough information in the 16 band audio tool to reconstruct any useful audio. 16 frequencies at 60Hz (60 fps) is only enough information to reconstruct 960 samples per second (16*60)-- you'd get a decent range, but you'd be missing tons of in-between tones due to the limited frequency bands and frame rate.
good point on AM - that's very true. (not sure if "blurs" are strictly AM, but the general idea is much better overall :)
I've honestly never heard of NI Reaktor, but it looks interesting. I've seen a couple other audio shader things (OpenAL has a few extensions for something like it, not sure if any are official), but none of them have really become "the one" to use yet. Definitely something to keep an eye on though :)
Got around to trying this. Thanks for the example!
It works a charm, except it seems you have to cast the return value as vec4:
May I ask what kind of magic goes into the luminosity calculation? My guess is you're correcting for human perception, but where did you get the values?
Those are standard coefficients for luminosity described in ITU-R 601. formerly this was known as CCIR 601.
(I wrote about this like 6 years ago, incidentally -- http://softpixel.com/~cwright/programming/colorspace/yuv/ )
There's a brief discussion on it here: http://www.fourcc.org/fccyvrgb.php Note that Y in YUV or YCbCr is luma, so the Y equation is what you're interested in here.
Ooops, yes. Very silly mistake there. Sorry about that.
If you want opaque alpha, you'd do
return vec4(vec3(smoothstep(threshold, threshold + smoothness, lum)),1.0);a|x
This whole are is a bit of a nightmare actually. I did a commercial CIFilter project a while back where I was asked to do a range of colourspace-conversion filters. I spend ages trying to work out how to do it exactly right, before eventually realising he just wanted an effect, so it really wasn't important for it to be mathematically/perceptually/chromatically 'correct'.
a|x
Ah, that makes sense!
We ended up using After Effects for the difference matte since I didn't have this threshold method at the time. But I'm learning, so thanks for that.
Ah, of course. Good catch.
In any case, the smooth is useful in some cases I think, but I like the non-smoothed version best for my purposes, since the smoothed one produces some larger midtone areas. If I need anti-aliasing, I think I'll go with a slight blur.
sadly, this is one area where macs are in the stone age, as far as I can tell. Mix in gamma (which changed from 10.5 to 10.6), ColorSync (which does "automatic" color balancing, only there's no clear documentation as to where/when it automatically does it), and any image technology (GL, QC, QT, CI), and it's a disaster. [I harp on this all the time].
I found it much easier on linux, and haven't really bothered with it on windows -- on linux at least, if your math was right, you got consistent results (smokris and I did a hardware-accelerated video mixer app several years back that involved transitions from RGB to YCbCr).
Totally agree with your sentiments (and that's not even touching ambiguities in the specs...)
This ended up in completely the wrong place.
This method will be more efficient that a post-process blur though. Blurs are notoriously GPU-intensive, and should be avoided where possible for realtime stuff.
a|x
I made a soft iris transition (or wipe in video editor lingo) with this CI filter that has decent fps even on my dual G5. Even a low-res masking image like [8,8] pixels coming out of the CI Filter can serve as an image mask on account of QC's sprite smoothing routines. Thanks for the code toneburst. I'm wondering if I can use to use the smoothstep function to make Glow and Morph filters next.
NB this comp needs two images/videos to wipe b/w. Also uses NI Circle patch (which gets installed when you demo the Noise Industry FCP plugins I think).
Yeah, I know.
The thing is that the smoothstep seems to have a tendency to leave in the slightest hint of midtone in some cases, which would look make a difference matte unusable.
So for processing this as a vfx while editing, I'd anti-alias with a blur. I'm talking sub-pixel dimension up to max 10 pixels.
For real-time, I'd probably use no smooth / antialiasing if it proved too expensive, or downsample the mask image before thresholding.
That's how anti-aliasing works :)
a|x
Heh, yeah, I should have written that it sometimes leaves larger contiguous midtone areas, whereas I only want the edges of 100% black areas antialiased. But I hear you ;)
Just picked up on this rather late - see Core Image Kernels for the link I meant to point you in the direction of.