Photoshop Channels

blueroom9000's picture

Hi QCCommunity, I'm quite new to Quartz Composer and I struggle to split a video source into its RGB channels (as I would do when using the Photoshop "Channels" pannel). I already searched a lot of forums, but didn't find the right patch to do so. Does anyone of you have an idea on how to separate the 3 channels of a video input. Do I need to dive deeper into core image programming? Thanks a lot! markus

cwright's picture
Re: Photoshop Channels

Core Image Filter

kernel vec4 redChannel(sampler image)
{
   vec4 color = sample(image, samplerCoord(image));
   return vec4(color.r);
}

Replace .r with .g or .b for green and blue channels, respectively.

If the alpha is giving you a hard time, do vec4(color.r, color.r, color.r, 1.0)

psonice's picture
Re: Photoshop Channels

Indeed, and you'll need 3 of these filters (one for each colour). Recombine them with:

kernel vec4 redChannel(sampler r, sampler g, sampler b)
{
    float r = sample(r, samplerCoord(r)).r;
    float g = sample(g, samplerCoord(g)).g;
    float b = sample(b, samplerCoord(b).b;
    return vec4(r, g, b, 1.0);
}

It's well worth learning a bit of Core Image Filter magic though. Depending on what you're doing, it might be very easy to do your processing inside a single filter, and not need to split / combine at all.

blueroom9000's picture
Re: Photoshop Channels

Thanks a lot, this helps me already a lot!! But how can I have the 3 channels as separate output? Can you recommend a source to get into Core Image Programming? Greetings from Germany, Markus

cwright's picture
Re: Photoshop Channels

unfortunately, CI only allows a single image output per filter. As such, you'll need 3 filters (pass the same image to each one) to extract 3 channels.

gtoledo3's picture
Re: Photoshop Channels

Chris, I know this isn't exactly the same line of question as multi channel output, but, as you know, core image filters can be programed that take advantage of the iterator. The main one that comes to mind is the depth computation developer example.

Some operations with stock image filters get really pricey when I program them to do something per iteration; one that comes to mind is blur. Are there any good rules of thumb that I may not be thinking of, to get better performance?

For example, when I place something like a blur inside of an iterator to make a stack of sprites get more blurred further back in Z, I'm usually using an iterator set to patch time external, connected to Current Position on the Iterator Variables. Should I be doing something Index based instead? Does it matter (I know, test test). However, stock CI patches deliver really inconsistent results, and this is one area that there is little documentation on, because of the fact that it's an intersection of Core Image and QC Iterator.

If I was to want to write a blur kernel (or any kernel with a "level") to blur per iteration, what is the "correct" way to do this? If you could explain how to do something like a per index level with CI, and maybe how to do call a curve (like a sin or cosine) to curve levels per index, it would be really informative. I don't know if you've ever thought about that kind of CI use or not.

cwright's picture
Re: Photoshop Channels

gtoledo3 wrote:
Some operations with stock image filters get really pricey when I program them to do something per iteration; one that comes to mind is blur. Are there any good rules of thumb that I may not be thinking of, to get better performance?

Do the blur up front? seriously, there aren't many filters more expensive than a blur (esp a good gaussian or disc blur).

gtoledo3 wrote:
For example, when I place something like a blur inside of an iterator to make a stack of sprites get more blurred further back in Z, I'm usually using an iterator set to patch time external, connected to Current Position on the Iterator Variables. Should I be doing something Index based instead? Does it matter (I know, test test). However, stock CI patches deliver really inconsistent results, and this is one area that there is little documentation on, because of the fact that it's an intersection of Core Image and QC Iterator.

QC iterator is irrelevant here -- it's simply causing a filter to get executed several times per frame. This isn't free, and scales quite poorly. You can use a cheaper blur, or pre-calculate the blurs up front. There aren't really any other tricks.

gtoledo3 wrote:
If I was to want to write a blur kernel (or any kernel with a "level") to blur per iteration, what is the "correct" way to do this? If you could explain how to do something like a per index level with CI, and maybe how to do call a curve (like a sin or cosine) to curve levels per index, it would be really informative. I don't know if you've ever thought about that kind of CI use or not.

I've not. Blurs are trivial complexity-wise (run a 2D kernel over the image) -- unfortunately that sucks performance-wise (O(N2M2)). some are nicely separable, which means they're "only" O(N2M). CI's already separated filters that are trivially separable (box blur, perhaps gaussian blur?). The down side of separable filters is that they take 2 passes (so you pay 2x the setup time -- usually this is "free" compared to the complexity of not separating, but it does take up a bit more memory). CI also employs other tricks where it will down-sample images for blurs, and compose blurs out of several partial blur steps, some of which are separable. These are all pretty clever and interesting, but at the end of the day it's imperative to understand that blurs cost a ton, and shouldn't be done live ever unless there's no alternative.

Writing your own blur in CI, you're never going to get the performance that the built-in blurs have, because they can do things you can't (like composing multi-pass filter stages -- you can kinda do that with the filter function, but not everything is immediately possible through that route). Using a "cheat" like Vade's mipmap blurs is probably about as good as you can get in the context of QC.

gtoledo3's picture
Re: Photoshop Channels

Thanks for the breakdown, it pretty much answers my question in my assumption that this isn't really a thing that's been fleshed out much.

For clarification, I really shouldn't have said "blurs". That's one thing I have done, but I should have stated "in general". I guess I was wondering if there was any good practice in getting a kernel to "ramp up" in effect per iteration, other than to attach an interpolate with time base to external/iterator variable combo to whatever parameter you wish to iterate.

There are several patches that work well in an iterator which also process image, such as image texturing properties.

On the other side of that, I think I've had others lock up completely (maybe constant color?). This makes me wonder about the possibilities of using the iterator for per index based C.I stuff.

I already understand the scenario of built in blurs vs. the core image kernel, but it's good to expound on that for record. I understand the nature of blurs as far as doing your own with CI kernel (or GLSL in QC) vs. the function of the built in ones. I'm also of the opinion that all image processing should be done ahead of time if it's prepared art. However, if something is generative, this isn't possible. I'm not trying to be blindingly obvious, but the reason I would use this method for with the iterator is to be able to control a dynamic number layers in real time with the furthest always being the most blurred (or most _whatever. Most : insert CI effect here).

vade's picture
Re: Photoshop Channels

I'm not technically doing mipmaps, since most images in QC are GL_TEXTURE_RECT, I just do really aggressive downsampling in the first multi pass stage and then "ramp" the resolution up each pass so the blurs get finer.

I've done some more 'correct' (read, not "cheating") blurs for NI that seem to be as fast as Apples Gaussian (which I know is highly optimized) by doing exactly what you have described, separable filters, and doing some tricks in the Obj-C side. But yea, what Chris said. Blurs (especially correct, good looking ones) are really heavily expensive. Avoid if you can.

cwright's picture
Re: Photoshop Channels

gtoledo3 wrote:
I guess I was wondering if there was any good practice in getting a kernel to "ramp up" in effect per iteration, other than to attach an interpolate with time base to external/iterator variable combo to whatever parameter you wish to iterate.

That depends on what "ramp up" means. For some effects (Saturate! Noiseify! Crossfade!), ramping has no cost (e.g. mixing 50% of 2 images costs the same as 25% of one and 75% of another -- the ramp as the percentage has no effect on the computational cost). Others, like blurs, increase horrifically in computational complexity as their "ramp up" increases (blur radius = 1 is free, radius = 2 is pretty cheap, radius = 4 is not 2x as difficult as radius = 2, it's 4x more expensive. radius = 8 isn't 4x as expensive as 2, it's 16x as expensive, etc -- math may be incorrect, but I'm just trying to show the idea). There's no hard and fast rule for how a given filter will scale, unfortunately.

gtoledo3 wrote:
However, if something is generative, this isn't possible. I'm not trying to be blindingly obvious, but the reason I would use this method for with the iterator is to be able to control a dynamic number layers in real time with the furthest always being the most blurred (or most _whatever. Most : insert CI effect here).

Even for generative stuff, cheats can be employed (for example, if you overlay a mask on an image, you can pre-calculate a blurred mask when compositing). There's no end to the number of tricks you can pull -- some will give identical results, and some will only give approximations, but that's just how it is unfortunately.

cwright's picture
Re: Photoshop Channels

vade wrote:
I'm not technically doing mipmaps, since most images in QC are GL_TEXTURE_RECT, I just do really aggressive downsampling in the first multi pass stage and then "ramp" the resolution up each pass so the blurs get finer.

Mipmaps are supported on rectangles, just not RECTs (as you've noted). Check out GL_ARB_texture_non_power_of_two at http://www.opengl.org/registry/specs/ARB/texture_non_power_of_two.txt. It was approved in 2003, and is supported by high-end graphical processing apparatuses such as my top-o-the-line GMA950. I kid you not. 3 of the named contributors are people I've talked to in the past week (and two of them have incidentally corresponded with me via email today for some work we're doing). It's kinda weird meeting these people, to be honest ;)

TL;DR version:

GL_ARB_texture_non_power_of_two wrote:
When this extension is supported, mipmapping, automatic mipmap generation, and all the conventional wrap modes are supported for non-power-of-two textures

Just saying. I hear the NPOT == No Mipmaps argument all the time, and it's about as antiquated as texture combiners. It's a shame that this doesn't drop-in work for QC :(

vade wrote:
I've done some more 'correct' (read, not "cheating") blurs for NI that seem to be as fast as Apples Gaussian (which I know is highly optimized) by doing exactly what you have described, separable filters, and doing some tricks in the Obj-C side. But yea, what Chris said. Blurs (especially correct, good looking ones) are really heavily expensive. Avoid if you can.

I was talking with the CI team lead last week about how disc blur works (disc blur is used for true depth-of-field. it's also laugh-out-loud expensive), and how they've optimized it (by slicing the kernel into pieces, some of which are trivially separable, and compositing on the pieces that aren't, but are now cheaper because they have sparse kernels or something), and my brain unscrewed the hinges on my skull and jumped out of my head onto the floor. It's that crazy. I'm glad I'm not on CI ;)

cwright's picture
Re: Photoshop Channels

(but to clarify some: the rectangle thing is kinda weird, and QC doesn't seem be designed to handle it as-is -- it has code that enforces texture dimensions to be powers of two for 2D textures)

gtoledo3's picture
Re: Photoshop Channels

... to that second part, calculating a blur mask is pretty expensive too.

Thanks for the feedback.

gtoledo3's picture
Re: Photoshop Channels

I notice that when I use Image Texturing Properties set to 2D, and the Mipmapping on it, I don't have any kind of power of two limitation on size for it to work or not. However, to my memory this feature didn't work at all in QC on my X3100/Macbook/Leopard, but works fine on the 9400/9600 nVidia/MacbookPro/SL.

It's really easy to tell that it's working in this scenario on non-power of two texture sizes, just by flipping mipmapping on and off; bit visual difference.

psonice's picture
Re: Photoshop Channels

Assuming your layers are static images, this is actually pretty easy to get working at full speed even with a high quality gauss blur :) All you do is pre-blur the images at various levels (do this outside of the iterator of course) and stick the blurred images into a structure.

The structure index is now your blur (or whatever) level, and inside the iterator you just use an index member patch instead of the blur. I've used this method to do really cheap DOF effects using sprites (I select the blur level based on the Z position of the sprite, and select the pre-blurred image.. 100 large sprites in realtime with gauss blur is no problem).

If your layers aren't static, you have to consider how to move as much as possible outside the iterator. If that's not enough, your only option is to optimise like hell. One thing I've done before is to use a much lower res image for any heavy blurs - instead of a 720p texture and a 20 pixel gauss blur, maybe you can downscale to 160x90 and do a 3 pixel blur. The result often looks pretty much the same, and you get a massive speed boost, despite the additional steps. Also put in some logic to remove any processing steps if they aren't used, like say if a layers is getting a blur of <1px.

gtoledo3's picture
Re: Photoshop Channels

I have a feeling I've done a poor job explaining my question, because I'm not asking about traditional methods of optimization. I was going to touch on the blur downsampling (eg., making the source image lower rez) in response to Chris above, but again, my question wasn't about blur.

I'm going to phrase my line of thought/question again, hopefully in a more clear way because I feel like no one has quite gotten what I really mean.

I can have a CI filter perform "more" to "less" of a function in an iterator, by connecting an interpolation to it, setting interpolation timebase to external, setting desired start/end values, and then iterating that value over the range set by the iterator. This is something one has to do with stock CI filters if using them in an iterator.

However, one can legitimately write a CI filter that is designed to work in an iterator.

kernel vec4 filter(sampler depth, sampler image, float plane, float resolution) {
vec2 coord = destCoord(); vec4 value = sample(depth, coord); float result = (value.x<=plane+resolution/2.)?((value.x>plane-resolution/2.)?1.:0.):0.; return result*sample(image, samplerCoord(image)); }

In the kernel above, some of the inputs are meant to work with Iterator Variables, on a per iteration level. Resolution is controlled by being fed an input value that is 1/Iteration Count (calculated w/ math and iterator var.)

The "plane" input is generated by dividing current index by iterations from the Iterator Variables patch.

This was my line of questioning. If one wants to use CI to effect per iteration, should I be thinking of writing kernels that are actually designed to "do cool stuff" in said iterator? I have a feeling this isn't something people have played with much.

("Q:What's the best way to play back 1000 sprites with different images in QC? A: play a movie of 'em!" "Q:What's the best way to setup a 3D scene in QC? A: take a screengrab of someone playing Call of Duty!" "Q:How do I make a hit map in QC? A: Load OpenEmu, you can play nintendo games!" (exit w/ charlie brown squiggle over head). ;-)

psonice's picture
Re: Photoshop Channels

Yeah, if you have to run the CI filters inside the iterator, use the iterator variables and design your filter to use it of course. Just keep in mind that might imply doing some extra maths to get the numbers in a useful range, so watch where you do that (you don't want to do it per pixel, sometimes it's still better to handle that outside of CI). Also, use the 'position' output from iterator variables, it's equivalent to index/iterations :)

Still, doing heavy lifting like CI filters inside the iterator is something to avoid if you can. Doing 100 iterations where you're processing a normally trivial filter on a normally trivial 640x480 image is equivalent to running that filter on a 6400 x 4800 image, plus a ton of overhead. If you have a case like that, pre-process the images outside of the iterator, then you just draw to screen.

gtoledo3's picture
Re: Photoshop Channels

As far as the position thing goes, that's on Apple (describing their patch arrangement/sample code) ;)

If you want to make an image queue where every next sprite gets "more" whatever, you have to use an iterator. All the pre-process stuff is n/a. Thanks for the thoughts.

psonice's picture
Re: Photoshop Channels

The only time you can't pre-process your images is if the source images are changing (i.e. video or something).

If there's a long list of images to process and you need quite a few steps, add a "loading" section to your comp, and run through the list of source images / filter steps once, feeding them all into the structure. It might be slow, but normally you don't care about start-up speed, you care about running speed so it's not important.

I've done this before for my demo stuff, where I created a colour font by running through the ascii table, creating and filtering a single letter as a sprite during each frame, then pushing them into a structure. Later I can make nice scrolling text with fancy fonts, no need to do everything per-letter, per-frame inside the iterator where the scroller is made :)

gtoledo3's picture
Re: Photoshop Channels

Cwright's post led me to a useful, non core image kernel based rule of thumb (maybe?)

Cwright's line of reasoning about certain things, like color shift, not being as hard as other effects, and the reasons why, and some more of his reasoning made me realize that setting whatever effect you want to the "maximum" desired level, and iterating a dissolve value instead, seems to work much better for effects where computation isn't linear. The visual result isn't quite the same, but it seems useful.

The discussion about disc blur below, and other points, show how vexing CI is in many ways, because the Core Image kernel can't necessarily "do" many of the CI effects in the way they are actually implemented by Apple.

vade's picture
Re: Photoshop Channels

Yea, I know about and use NPOT MipMaps in my ASSIMP based 3D model loader Im working on, but since pretty much everything is RECT from public API land (and has to be when you provide an output image from as a texture) I don't use them.

As far as the disc blur, I got a great WWDC walkthrough on it, and am trying to implement a similar semi-separable composite/ shaped one for NI. But man, those guys are badass, and Im a bit stuck, ha. Do you know who you spoke to? I forgot to get a card! Argh :)

cwright's picture
Re: Photoshop Channels

vade wrote:
Yea, I know about and use NPOT MipMaps in my ASSIMP based 3D model loader Im working on, but since pretty much everything is RECT from public API land (and has to be when you provide an output image from as a texture) I don't use them.

I'm glad you knew -- I honestly had all the pieces floating around in my brain, but I didn't put them together until just a few days ago. I wish someone would have just said "GL_TEXTURE_2D without POT dimensions!" -- it would have saved me a few years of knowing-but-ignoring ;)

vade wrote:
As far as the disc blur, I got a great WWDC walkthrough on it, and am trying to implement a similar semi-separable composite/ shaped one for NI. But man, those guys are badass, and Im a bit stuck, ha. Do you know who you spoke to? I forgot to get a card! Argh :)

Yep, I know who I talked to ;) They're currently pretty slammed with some work going on (as am I), but when things slow up I can try and see where things are.

psonice's picture
Re: Photoshop Channels

I'm not so sure it's a general rule of thumb like this.. a better rule of thumb is to pre-calculate the hard stuff as much as possible if speed is going to be an issue. That's pretty much all I've been saying. Do the heavy lifting outside of the iterator and replace it with something light. I mean hey, if all you're doing is blurring some static images, do a really high quality blur in photoshop and just import the images, you'll get better speed AND quality :D

I've certainly seen why this is important recently, the cone mapping shader I was doing absolutely needs pre-calculating. It takes ~5 seconds to generate the cone map for a tiny 128x128 image (and this is not yet at full quality, it'll probably be 2-4 times slower once it's finished!) But that only has to be done once, and the final shader runs at full framerate because the heavy calculation is already done. Without the precalc you're looking at a few frames per minute :O

gtoledo3's picture
Re: Photoshop Channels

In a moment of "doth protest too much" about my "I'm not talking about just blur in an iterator, I swear!", I'm posting a composition that shows blur w/ an iterator in a way that can't be done by "pre-blurring art" etc.

...but any CI filter, or filter+dissolve can be setup to do this kind of thing.

The key to OK performance is to make sure the source image going into the queue is fairly low pixel count.

PreviewAttachmentSize
Iterator - Blur_With_Distance.qtz12.78 KB

psonice's picture
Re: Photoshop Channels

Yeees, except that would be fairly trivial to do with a pre-computed blur, and it would run much faster :D You'd just make say 10 copies of the sprite image with 10 levels of blur, then handle all of the sprites inside the iterator (or in javascript or a fancy queue or whatever for even more speed).

I think I get where you're coming from though. Try replacing that simple sprite animation with a video input and it becomes impossible :)

A more important rule of thumb (mainly a programming one, but it applies plenty to QC): optimise the algorithm before you optimise the implementation. You'll get small gains from optimising the way you do the CI filters, but potentially massive gains from rethinking the way your whole setup works.

gtoledo3's picture
Re: Photoshop Channels

Yeah, if the content is being created on the fly, and layer count can be variable, then the whole pre-compute thing is not a possibility. I've done the thing of "pre"treating image and then sending image structures through the iterator, but it won't produce the same result for certain content (there is a multi-channel image queue example on my website that does this).

If there's a performance difference between plopping 10 sprites out, taking a structure index off of a queue, feeding to 10 separate blurs, etc. and doing the same thing with an Iterator, which is the only way for it to be fully dynamic in object count, then it's a problem of iterator design, not because I'm inherently doing anything "wrong". However, I don't tend to think that the iterator is making anything worse (at least in my tests). If you blur 10~20 decent size images with a CI blur, things are going to suck anyway you cut it.

I, of course, have to deal with reality of how to optimize (eg. if iteration is more taxing than not using iteration), but it's not as if the thought of iterating CI is inherently wrong, especially considering it's intrinsic to many Developer examples (not saying you're suggesting this, I'm just stating this opinion).

I know what you mean about optimizing the algorithm, as far as being able to make concessions with CI filters that really speed stuff up tremendously. I guess one of my "jags" in this iteration/CI thing is that since CI source isn't always available (err, rarely at all), one has to go scrounging to reimplement filters to make them work faster.

This is kind of OT but sort of relates; yesterday I stumbled on a way of doing gradient map blur stuff (eg., tiltshift, DOF-emulation type stuff) extremely quickly, but it falls apart at higher levels of effect (as opposed to a more legitimate route). I would use it for realtime stuff extensively, but probably wouldn't use it for offline render. I'm of the opinion that there usually isn't one right way to do something, it tends to depend on context, and each approach has positives and negs (eg. looks good vs. performance, ease of use vs. performance, etcetera).

psonice's picture
Re: Photoshop Channels

Here's a version using pre-computed blurs. Slightly different, but pretty much the same effect (it's possible to make it identical). Runs fine with 200 iterations :)

PreviewAttachmentSize
Iterator - Blur_With_Distance2.qtz17.36 KB

psonice's picture
Re: Photoshop Channels

Quick reply as it's time to go: The source for the CI filters IS available! You need to find the actual CI filter on disk (can't remember off-hand where they're stored, cwright could probably tell us but probably won't in this case :D) Open it in a text editor (going from memory here, been a while since I did it), it's binary but the filter itself is there in text form. It'll take some figuring out, but it's worth it in some cases.

Btw, I have a passable variable blur filter here somewhere if you want it - it takes a source image plus a separate blur radius image (i.e. depth map for DOF, gradient for tilt shift).

gtoledo3's picture
Re: Photoshop Channels

I see where you're going with that, and for the composition as submitted, it makes sense (though as you say, not timed correctly). The thing that's key for me is being able to freeze an image structure, then manipulate the effect curve, not just render stuff that's been effected. So, I could "mouse" a landscape dynamically, freeze scene, then control blur (or cartoon effect, or b/w, etc) from back to front/front to back.

The way that you've jiggered that is particularly useful for some scenarios though, for sure! This effect can add depth to a scene, and the bonus, is that if the objects are smallish, if you render it in 3D (eg. polarized or anaglyph), you don't really notice that they're flat at all.

gtoledo3's picture
Re: Photoshop Channels

Oh snap, I forgot about that. Already have it (cwright took no part in this, had it for ages, at least circa Leopard!).

It is worth mentioning that the blackop page on QC has some pretty decent CI examples, for those interested in some openly posted CI code that shows a good array of techniques. http://b-l-a-c-k-o-p.com/Qamera/Quartz-Composer/ I learned a decent amount from that page.

I would be interested in seeing your variable blur. Someone here posted a very good one that is somewhat taxing on performance, but can be used realtime if the scene isn't heavy. NI has one that I have no experience with, but that I've eyed in my patch list. The fellow that did the ARToolkit made a plugin that does it, and isn't bad, but it's extremely slow. I have my own that is extremely quick, but is also cheap and will start to look like the image is layered/stacked at areas of extreme blur but, used correctly it is passable and adds almost no overhead. I would be interested in seeing one that's really good when working with depth maps. I guess there is a challenge in getting good depth map info out of QC, but that's another story...

vade's picture
Re: Photoshop Channels

Try the NI ones. They should work fast (all are separable), have a variable pass amount so you can control the granularity. They are pretty decent. The pure GL ones are very fast.