Low / high pass filters

sbn..'s picture

Hello all,

For a composition I'm doing, I need something akin to a clamp or high pass filter. What I mean is a patch that'll accept an image and a color input, and then output a pure black / pure white image using the color as a threshold.

For my application, this would be used with an image that's already b/w, so it could use e.g. only the red channel.

I've tried my hand at a Core Image filter that does this, but I'm utterly bewildered since the documentation is so scant. I keep getting errors about data-dependent conditionals and swizzles.

Is it really impossible to return a value from a CI filter based on a conditional statement?

Or is there an obvious way of doing this I'm missing? In other software, I'd use a curve adjustment or similar, but I haven't found an equivalent in QC. The color controls is too fidgety and dependent on the input image.

Any pointers would be appreciated.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

usefuldesign.au's picture
Re: Low / high pass filters

Cwright posted this comp recently on the Apple QC list so I'm assuming it's legit to post it here. It demonstrates use of conditionals inside a CI filter. I agree, the Apple documentation is scant/quite abstract in nature for non comp-sci types like myself. More examples would be good, I tend to learn from pulling examples apart as much as those kind of docs.

PreviewAttachmentSize
CI arc-1.qtz3.79 KB

toneburst's picture
Re: Low / high pass filters

You just need something like this really:

kernel vec4 threshold(sampler image, float threshold) {
 
   // sample input image at working pixel
   vec4 px = sample(image, samplerCoord(image));
 
   // get pixel luminosity
   float lum = dot(px, vec4(0.299,0.587,0.114,0.0));
 
   // Return black or white
   return (lum > threshold) ? vec4(1.0) : vec4(0.0,0.0,0.0,1.0);
}

threshold will be in the range 0.0 to 1.0.

The key is the use of the ternary (?) operator instead of the more traditional if( ) { } conditional syntax.

a|x

cybero's picture
Re: Low / high pass filters

Nice example toneburst, I find that I have to set the LFO just above 1 [range 0 to 1.1 or offset 0.1] to obtain a longer term black on the pass. Interesting tip about the ternary operator.

Regarding tutorials for CI, sbn, http://cycling74.com/2007/05/23/your-first-shader/ is as good a place to start as any, although actual code snippets and examples are very useful, nonetheless.

toneburst's picture
Re: Low / high pass filters

Thanks for the link cybero- I'll check that out.

Incidentally, the code above will produce aliasing, since it's binary (just black or white output). You could use smoothstep() to smooooth things out a bit.

kernel vec4 threshold(sampler image, float threshold, float smoothness) {
 
    // sample input image at working pixel
    vec4 px = sample(image, samplerCoord(image));
 
    // get pixel luminosity
    float lum = dot(px, vec4(0.299,0.587,0.114,0.0));
 
    // return black, white and gradated grey in between
    return smoothstep(threshold, threshold + smoothness, lum);
}

smoothness can now be used to smooth out the edges of the result.

a|x

toneburst's picture
Re: Low / high pass filters

Oh, just realised- I've looked at that Cycling74 page before.

That's GLSL, so won't quite work as-is in a Core Image Kernel/Filter patch. Althought GLSL and Core Image Filter Language are similar, only GLSL Fragment shaders are roughly equivalent to Core Image Filters, so you might find a lot of tutorial material on GLSL covers a lot of stuff CIFilters can't do, and therefore isn't strictly applicable to CI.

That said, there are a some simple GLSL fragment shaders out there that can relatively easily be converted to Core Image filters.

In terms of actual CIFilter documentation, I never really managed to find much, even on the Apple site. There are certain aspects of CIFilter operation I've never fully got my head around.

a|x

sbn..'s picture
Re: Low / high pass filters

Thank you all for the replies.

It turned out to be a bit of bad timing on my end, I've been putting out fires at work and had to put this on hold for a bit.

I had this idea for a difference matte implementation, but just ran in to one of QC's blind spots as often happens. Really, low / high / band filters would be most useful.

Especially the examples are extremely useful. Much obliged. I can see I need to read up on the ternary op, as well as look at the tuts.

All in all, thank you.

toneburst's picture
Re: Low / high pass filters

No prob.

It took me a while to get my head around ternary conditionals. I find myself using them in JavaScript, now too, though, since they're neat and compact.

a|x

itsthejayj's picture
Re: Low / high pass filters

Had a little play with this one... think you might like ;)

PreviewAttachmentSize
CICircle3.qtz9.48 KB

toneburst's picture
Re: Low / high pass filters

That's nice- with antialiasing. Cool. I can see potential there. I might do a multi-pass version of that, I think.

a|x

toneburst's picture
Re: Low / high pass filters

Incidentally, it's funny you should talk about 'high/low-pass filters' in this context. I'm guessing, like me, you come from an audio synthesis background. High and low-pass filters mean something quite different in image-processing terms, but I knew exactly what you meant because I've used subtractive synthesis methods in audio a lot over the years.

It's funny how terminology crosses-over between disciplines (and sometimes doesn't).

a|x

cwright's picture
Re: Low / high pass filters

perhaps I'm rusty, but high-pass/low-pass (and band-pass) means the same thing in audio that it does in image processing -- audio is 1D while images are 2D, but the principles of filtering are the same.

In the OP's post, it was more of a thresholding problem (nothing to do with actual band filtering).

low-pass-filtering an image means removing the high frequency components (the "detail" so the image looks blurred sort of), while high-pass-filtering removes the low frequency components (the slowly-changing portions, so the image looks like an edge-detected variant of sorts). - http://local.wasp.uwa.edu.au/~pbourke/miscellaneous/imagefilter/ (pbourke, as always, is prolific ;) -- neither of these are taking place in this context.

Perhaps audio synthesis is different, but if so, I'm inclined to say they're "wrong", as high/low pass filtering hasn't meant anything other than the above for a century or two for both 1- and 2-D data sets (and more)

toneburst's picture
Re: Low / high pass filters

All that is true cwright, but it's the way the terms are being used here implies an equivalence between luminosity (in graphic terms) and frequency-content (in audio terms), whereas, as you rightly point out, in strict image-processing terms, hi and low-pass filtering apply to the low and high-detail areas of the image, rather than areas of differing intensity.

I'm hazy on the details of all this (and much less than hazy on the mathematics behind it), but I do like the idea that if you analyse images in the frequency domain, it's possible to apply a lot of the same processes used in digital audio manipulation. Actually, some way of converting audio>video>audio in QC would be really cool, thinking about it.

a|x

cwright's picture
Re: Low / high pass filters

(note that I'm not trying to be a hard-nose or anything)

For operations such as this, where no frequency-domain (or even time-domain) transformation takes place, using the term "high pass" or "low pass" is misleading at best -- each pixel is evaluated entirely independently of the others. A more fitting term would be thresholding.

Converting audio to/from images is something I need to talk smokris out of every few months. I do this because, while it sounds incomprehensibly awesome and amazing, it's actually quite lame; let me 'splain.

Every conceivable operation that you do on an image can already be done on audio. Every image from audio would be 1 pixel tall (or 1 pixel wide), which makes most filters stupid. If you make non-1D images, you'll get periodic effects that may be interesting, but are still better served by actual audio processing.

There's also the overhead of transferring a chunk of data to/from GPU. This overhead is awful for audio due to the latency involved.

The only really interesting aspect of this is the idea that you could roll your own filter, and have it be a makeshift fancy audio filter. I contend that a much better solution to this would be an "audio shader" type function, where you can do per-sample programmable operations. Audio Units kinda do this (as do VST plugins), but having a runtime language rather than a compile-time language would make it much more interesting.

toneburst's picture
Re: Low / high pass filters

cwright wrote:
(note that I'm not trying to be a hard-nose or anything)

Didn't think you were, don't worry.

Quote:
For operations such as this, where no frequency-domain (or even time-domain) transformation takes place, using the term "high pass" or "low pass" is misleading at best -- each pixel is evaluated entirely independently of the others. A more fitting term would be thresholding.

Absolutely. Exactly the point I was making, in fact.

Quote:
Converting audio to/from images is something I need to talk smokris out of every few months. I do this because, while it sounds incomprehensibly awesome and amazing, it's actually quite lame; let me 'splain.

Every conceivable operation that you do on an image can already be done on audio. Every image from audio would be 1 pixel tall (or 1 pixel wide), which makes most filters stupid. If you make non-1D images, you'll get periodic effects that may be interesting, but are still better served by actual audio processing.

Not quite. If you do an FFT on the audio, you can represent each frequency-band of the audio signal as a 1D row (or column) of a 2D image. There are quite a few audio applications that allow audio>'frequency print'>audio resynthesis, with various image-based editing/mangling tools that can be applied to the intermediate stage. Years ago, there was an application called Metasynth that was based around this idea, but it's also been used in audio-restoration software for a while, too.

Quote:
The only really interesting aspect of this is the idea that you could roll your own filter, and have it be a makeshift fancy audio filter.

That's the kind of thing I had in mind, in my simple-minded way.

Quote:
I contend that a much better solution to this would be an "audio shader" type function, where you can do per-sample programmable operations. Audio Units kinda do this (as do VST plugins), but having a runtime language rather than a compile-time language would make it much more interesting.

That's a nice idea. I'm thinking another C-like language, maybe called something like AUSL.

a|x

cwright's picture
Re: Low / high pass filters

toneburst wrote:
Not quite. If you do an FFT on the audio, you can represent each frequency-band of the audio signal as a 1D row (or column) of a 2D image. There are quite a few audio applications that allow audio>'frequency print'>audio resynthesis, with various image-based editing/mangling tools that can be applied to the intermediate stage. Years ago, there was an application called Metasynth that was based around this idea, but it's also been used in audio-restoration software for a while, too.

When you generate a 2D FFT from a 1D audio source, you're "cheating" (i.e. you need to do windowing) -- windowing requires working with groups of samples, so there's bleed-over between neighbouring rows. There's nothing wrong with this, mind you, and it's used quite successfully as you've noted. However, it's not a straightforward lossless operation, iirc (it's been half a decade since I've studied/used digital signal processing though, so I'm definitely out of practice on this)

If you just do a naive audio-samples-to-luma image generation, operating on the resulting image doesn't serve much of a point. Doing a frequency-domain image would definitely be useful though (in fact, I don't know why I've overlooked this aspect, as it's quite obvious and frequently done...) Thanks for noting it :)

sbn..'s picture
Re: Low / high pass filters

Heh, what am I thinking.

Of course you're right about the terminology. I even knew this had I just taken the time to think it over.

Yes, I've dabbled in audio before (Reaktor, Max, Buzz), but it's been a while. Maybe the audio way of thinking is more ingrained than I thought. These days I work in images exclusively.

ETA: That goes for both of you, thanks for correcting the terms. And for he interesting discussion.

Yes, what I was asking for was simple thresholding. In a broader sense, what would one call this sort of operation, if we included the more advanced "levels" and "curve" operations? Value-mapping?

If it were extended so that it would pass values above/below the threshold unchanged. Value clamp?

If it had two thresholds and would pass whatever was between the thresholds. Dual clamp?

It's this sort of thinking that got me thinking about filters. And while I accept that it would only confuse matters to call it that since there's already a use for that word in imaging, in any other sort of programming I'd call that sort of operations "filtering" e.g. a list of numbers even though no frequencies are involved. Hmm - maybe I should try to change my thinking and call that "mapping" or "clamping" instead.

cwright's picture
Re: Low / high pass filters

if you're curving it, but there's a 1:1 correlation, it's mapping (there's possibly another term for this, but I don't know it off the top of my head).

with selective passing (of low values, or high values, or values between two thresholds, or values outside of two values), it's called multi-level thresholding (and sometimes multiband thresholding, if you're working with multiple bands of data -- Red, Green, Blue, Alpha would be 4 "bands", for example).

in photography, "filtering" is generally used to affect mapping, but also does some non-per-pixel operations (diffusion and cross-screens, for example, do fancier effects), but they're still not filtering in the signal processing sense (i.e. there's no "low pass" filter in photography that I'm aware of, though it'd be pretty awesome :))

(disclaimer: I was a photo nerd a bit in high school)

sbn..'s picture
Re: Low / high pass filters

Good to know.

In the case of photographic filters (at the point of recording), I suppose you are filtering frequencies in a very real sense, attenuating some frequencies of light more than others.

BTW, since we're on the topic: What's your guesstimate on what's more efficient: Using the stock CI patch, or wrapping a CI object in a custom patch?

I ask because having basic threshold / mapping nodes would be neat. I'm unsure if it's more QC-like making separate patches (in which case a few macros would suffice), or a monolithic patch with modes for different value transformations. What do you think?

cwright's picture
Re: Low / high pass filters

regarding filter -- I suppose so, though no one thinks of light frequencies unless they're really into physics (high pass = UV passes, low-pass = IR passes) ;) so it's still a per-pixel type operation, not a "remove low frequencies from this image" operation. (semantics, I know, I know -- I'm not overly attached to either, just voicing how it "typically" is)

regarding optimal performance: do both, profile.

dust's picture
Re: Low / high pass filters

nice circle that would make a good high/low pass filter knob for an audio filter.

dust's picture
Re: Low / high pass filters

im taking graduate class in perceptions this semester and well i have only been in the class a week but it seems not only can image and audio filtering be applied to each other but psychophysics also uses these terms which seem to be interchangeable as well. psychophysics is basically the study of how external stimuli can effect us mentally. yeah it sounds a bit weird but thresholding, signal detection both indiscriminate, and discriminant, as well as multi-dimesnsional sampling techniques are used etc.. at least the principles of psychophysics hold true to both sound and vision if you use Fechners law. Sure the math is the same both in the audio and visual world but it ultimately comes down to how these audio visual filters effect us. Im still a little unclear of how we are going to measuring things like this but i defiantly know how a lowpass resonant audio filter can effect my stimuli internally.

toneburst's picture
Re: Low / high pass filters

dust wrote:
i defiantly know how a lowpass resonant audio filter can effect my stimuli internally.

Ha! I know what you mean. I'm a bit of a synth nerd, and those simple subtractive synth sounds get me every time.

a|x

toneburst's picture
Re: Low / high pass filters

cwright wrote:
When you generate a 2D FFT from a 1D audio source, you're "cheating" (i.e. you need to do windowing) -- windowing requires working with groups of samples, so there's bleed-over between neighbouring rows.

Ah, OK. I've heard the term, but was never sure exactly what i meant.

Quote:
If you just do a naive audio-samples-to-luma image generation, operating on the resulting image doesn't serve much of a point.

You'd essentially be doing AM (amplitude modulation) on the audio, I imagine. And probably massively downsampling it too, unless you were working with very long 1D images (that were different lengths per-frame, depending on framerate).

Quote:
Doing a frequency-domain image would definitely be useful though (in fact, I don't know why I've overlooked this aspect, as it's quite obvious and frequently done...) Thanks for noting it :)

That was what I had in mind. Should have made this clearer from the beginning. In fact, you could make some kind of 16-channel/band 'spectral-print' visualiser with QC's builtin tools, I think. The trick would be to resynthesise than back into audio after it had been modified in various ways.

Incidentally, I don't know if you've ever played with NI Reaktor, but it has the facility to code custom DSP modules. I've not investigated it, but I have a feeling it might be a proprietary version of something like the 'audio shader language' we were talking about. Just thought of that.

a|x

cwright's picture
Re: Low / high pass filters

unfortunately, there's not enough information in the 16 band audio tool to reconstruct any useful audio. 16 frequencies at 60Hz (60 fps) is only enough information to reconstruct 960 samples per second (16*60)-- you'd get a decent range, but you'd be missing tons of in-between tones due to the limited frequency bands and frame rate.

good point on AM - that's very true. (not sure if "blurs" are strictly AM, but the general idea is much better overall :)

I've honestly never heard of NI Reaktor, but it looks interesting. I've seen a couple other audio shader things (OpenAL has a few extensions for something like it, not sure if any are official), but none of them have really become "the one" to use yet. Definitely something to keep an eye on though :)

sbn..'s picture
Re: Low / high pass filters

Got around to trying this. Thanks for the example!

It works a charm, except it seems you have to cast the return value as vec4:

return vec4(smoothstep(threshold, threshold + smoothness, lum));

May I ask what kind of magic goes into the luminosity calculation? My guess is you're correcting for human perception, but where did you get the values?

cwright's picture
Re: Low / high pass filters

Those are standard coefficients for luminosity described in ITU-R 601. formerly this was known as CCIR 601.

(I wrote about this like 6 years ago, incidentally -- http://softpixel.com/~cwright/programming/colorspace/yuv/ )

There's a brief discussion on it here: http://www.fourcc.org/fccyvrgb.php Note that Y in YUV or YCbCr is luma, so the Y equation is what you're interested in here.

toneburst's picture
Re: Low / high pass filters

sbn.. wrote:
Got around to trying this. Thanks for the example! It works a charm, except it seems you have to cast the return value as vec4:

return vec4(smoothstep(threshold, threshold + smoothness, lum));

Ooops, yes. Very silly mistake there. Sorry about that.

If you want opaque alpha, you'd do

return vec4(vec3(smoothstep(threshold, threshold + smoothness, lum)),1.0);

a|x

toneburst's picture
Re: Low / high pass filters

This whole are is a bit of a nightmare actually. I did a commercial CIFilter project a while back where I was asked to do a range of colourspace-conversion filters. I spend ages trying to work out how to do it exactly right, before eventually realising he just wanted an effect, so it really wasn't important for it to be mathematically/perceptually/chromatically 'correct'.

a|x

sbn..'s picture
Re: Low / high pass filters

Ah, that makes sense!

We ended up using After Effects for the difference matte since I didn't have this threshold method at the time. But I'm learning, so thanks for that.

sbn..'s picture
Re: Low / high pass filters

Ah, of course. Good catch.

In any case, the smooth is useful in some cases I think, but I like the non-smoothed version best for my purposes, since the smoothed one produces some larger midtone areas. If I need anti-aliasing, I think I'll go with a slight blur.

cwright's picture
Re: Low / high pass filters

sadly, this is one area where macs are in the stone age, as far as I can tell. Mix in gamma (which changed from 10.5 to 10.6), ColorSync (which does "automatic" color balancing, only there's no clear documentation as to where/when it automatically does it), and any image technology (GL, QC, QT, CI), and it's a disaster. [I harp on this all the time].

I found it much easier on linux, and haven't really bothered with it on windows -- on linux at least, if your math was right, you got consistent results (smokris and I did a hardware-accelerated video mixer app several years back that involved transitions from RGB to YCbCr).

Totally agree with your sentiments (and that's not even touching ambiguities in the specs...)

toneburst's picture
Re: Low / high pass filters

This ended up in completely the wrong place.

sbn.. wrote:
If I need anti-aliasing, I think I'll go with a slight blur.

This method will be more efficient that a post-process blur though. Blurs are notoriously GPU-intensive, and should be avoided where possible for realtime stuff.

a|x

usefuldesign.au's picture
Re: Low / high pass filters

toneburst wrote:
This method will be more efficient that a post-process blur though. Blurs are notoriously GPU-intensive, and should be avoided where possible for realtime stuff.

a|x

I made a soft iris transition (or wipe in video editor lingo) with this CI filter that has decent fps even on my dual G5. Even a low-res masking image like [8,8] pixels coming out of the CI Filter can serve as an image mask on account of QC's sprite smoothing routines. Thanks for the code toneburst. I'm wondering if I can use to use the smoothstep function to make Glow and Morph filters next.

NB this comp needs two images/videos to wipe b/w. Also uses NI Circle patch (which gets installed when you demo the Noise Industry FCP plugins I think).

PreviewAttachmentSize
Soft Iris Wipe.zip597.38 KB

sbn..'s picture
Re: Low / high pass filters

Yeah, I know.

The thing is that the smoothstep seems to have a tendency to leave in the slightest hint of midtone in some cases, which would look make a difference matte unusable.

So for processing this as a vfx while editing, I'd anti-alias with a blur. I'm talking sub-pixel dimension up to max 10 pixels.

For real-time, I'd probably use no smooth / antialiasing if it proved too expensive, or downsample the mask image before thresholding.

toneburst's picture
Re: Low / high pass filters

sbn.. wrote:
Yeah, I know.

The thing is that the smoothstep seems to have a tendency to leave in the slightest hint of midtone in some cases, which would look make a difference matte unusable.

That's how anti-aliasing works :)

a|x

sbn..'s picture
Re: Low / high pass filters

Heh, yeah, I should have written that it sometimes leaves larger contiguous midtone areas, whereas I only want the edges of 100% black areas antialiased. But I hear you ;)

cybero's picture
Re: Low / high pass filters

Just picked up on this rather late - see Core Image Kernels for the link I meant to point you in the direction of.