Bugs/Special features

this is amazing :) I only run into bugs that are not useful... typing "while(1);" in Javascript, "atan2(x)" in Mathematical Expression, etc.

Probably the most annoying feature I've seen in a long time though! As soon as the parser can read the javascript without errors, it runs it - which is great, as you can do 'live coding', but it's near impossible to write a for loop live without making it an infinite loop and killing qc.

Stopping playback while you write it helps. And saving first ;D It gets me every time that one.

i never found any "really" usefull bugs. On the other hand, connect an animated image coming from a CI chain to a GLSL's sampler 2D... and you won't even go beach balling.

I've not tried that yet, what happens? Kernel panic? Does it still do it if you use a texture output from render in image?

not a kernel panic, thankfully, but it crashes QC. Basically, anything with a CoreImage anywhere in the chain will kill a sampler2D in the vertex shader. The Render In Image solution might be interesting to try... I don't know if I kept the demo around (I'm sure I did, I just don't know where... accumulating thousands of images, videos, compositions, and 3d models for some reason lately... ;)

it crashes in any case, esp. with a render in image. Strangely, when using my hacked HeightField plugin, i can distort a grid with CIimages, even Render2TEX. (disabling color correction DOES help, but not that much) GLSL must be Baaaadly coded... (skanky ?:)

I suggest that because I saw it has a choice of 'rectangle' or '2d' output, and as the other end is called 'sampler2d'... :)

That must mean that there's some step in the Official API's image handling stuff that 'flattens' a CIImage before shipping it off to GL for vertex displacement that doesn't get called in QC's unofficial API by default. If I'm understanding this correctly, that means it should be fairly trivial to fix this bug in GLSL with just a couple lines of code. (I don't think we can do it easily, since GLSL's a QCProgrammablePatch, which we don't understand very well yet).

Interesting note, thanks for that info franz! :)

That's pretty annoying, it's one of the patches I was hoping to (ab)use. Has anyone looked if it's on apple's Big List of Bugs?

it is a known bug. However, i never took the time to file a bug report (shame on me)... and it doesn't seem that many ppl use the GLSL patch...

For the handful of you not on the QC-Dev mailing list, today I uncovered a memory leak in Apple's Structure Sort patch. Basically, each time it sorts it leaks a tiny bit, and over time repeatedly sorting slows down the system.

Thankfully, Troy Koelling was on top of his game, and he found the issue within a day (sounds like he should work for Kineme ;):

From:   Troy Koelling
Subject:Re: Leaky Structure Sort
Date:   February 26, 2008 3:01:01 PM EST
To:   cwright

Thanks, I found the issue. Hopefully we can get this into a software update.

Pretty cool :)

Perhaps I'm misunderstanding here, as I'm new to GLSL (actually started lunch time today :) but it looks to be working here.

I have a render in image with a whole bunch of stuff in it producing an animated texture, which I'm feeding in to a GLSL patch through sampler 2d. I have a surface showing with an animated texture (and lighting soon if I can figure out the normals.. starting out with a grid and deforming it is a pain in the arse :)

Looks like they've made some attempts to fix it: now if you attach the image initially, it doesn't crash (but it doesn't displace either). But attach a normal image, and ten switch back to an animated one, and 'splode.

Here's a sample composition: Connect the video input to the Displacement input as well (where the Plasma Image plugs in), and you'll see it :) (be sure to save your work before hand)

[edit: glsl code shamelessly stolen from franz's crashing sample from december, 2007]

Messing about with the GLSL code while it's running gives you a fair chance of the same style of crash. It's like the while() thing with the javascript patch, but much much worse.

Still, if you stop the composition before switching it runs fine, so it's not an issue really if you remember to stop first (and save!). Unless you're doing a live performance of course, which I guess a fair few of the guys here are doing.

Not to be a total nerd, but that's not quite correct:

1) you can unplug the plasma, and reattach it. No harm, no foul. Everything works as expected. You also do this with static images from an Image Downloader / Image Importer without any ill effect.

2) Stopping and restarting does indeed make it not crash, but it also Doesn't Displace - I mentioned this as the "sort-of fix" previously. Working with Core Image by ignoring it doesn't count as "fixed" in my book :) Not fine :(

3) stop the composition. Plug the plasma image into a box blur, and the box blur into the displacement input. Start the composition. Crash. No switching involved.

Further, the GLSL code is not executing when the port changes; port changes happen between frames, when no patch is executing. The image application method (-[imagePort setOnOpenGLContext:options:]) takes place before any GL stuff gets activated. The bug is that it's not properly evaluating the CI image before setting it on the context. Non-CI images don't appear to do this.

The sampler2d seems to work fine as a fragment shader input, but not in vertex. Yet it works for the plasma. Very odd!

I've just had a play with my own composition, and got absolutely nothing from the texture in the vertex side. Could it be then that it's being passed the wrong pixel format, and the values being returned are always zero? The plasma looks like a prime candidate for an 8-bit image output, whereas the blur would probably push out 32.

The render in image patch has bpp settings, but I don't believe it does what it says on the tin. There's no colour banding or dithering if you render a colourwheel in it at 8bpp, and at 32bpp the output becomes pixelated. I bet it's 8bit per colour channel, then 16/32 bit float.

I like your thinking. I hadn't considered incorrect formats, but that got me thinking, there are a few other internals where QC doesn't handle the colorspace correctly (greyscale doesn't work, for example).

So I made another experiment: Instead of accessing based on the texture coordinate (which, if the format is wrong, could go out of bounds if there's not enough data -- Not sure why that would happen though, since 32bpc is much larger than 8bpc, but we'll assume it could), I set the texture coordinate in the sampler to vec2(0.,0.) in place of gl_MultiTexCoord0.xy-- unless it's getting less than a few bytes, this shouldn't ever go out of bounds.

This test also crashes. Sadly, this isn't the cause.

Secondary test: Loading an hdr (32bpc) image with the Image Importer and feeding that to displacement map would also cause problems if the GLSL patch doesn't handle >8bpc correctly.

This test does not crash or misbehave (it's difficult to tell how accurate it is without inspecting closer, but it looks about right after a few seconds of inspecting). So the GLSL patch does handle >8bpc correctly. The problem still lies in Core Image, from what I can tell.

I try to put coreimage to software and keep openGL to hardware

with the "OptionClic Preferences" in "editor" enable softwareCoreimage : now it seems to work ok with filters, blendingmode,and VIDEOS :) in the displacementMap input.... ;)

yanomano.

Preview	Attachment	Size
	CISoftware.png	186.22 KB

yanomano, you are my hero. Looks like my hunch (CoreImage sucks dot com) was sort of right, but sort of wrong -- it's not a framework bug, but a driver bug or something? Very weird. Happens on several different cards though; surely not all drivers have the same bug? Maybe it still is a framework bug, but at a low enough level that the software renderer safely bypasses it.

Thank you so much for discovering this :)

It works, but god it's slow! Lets hope apple come up with a proper fix soon, 15fps is a little too low for pleasurable viewing!

I was playing with the Image Pixel Patch (it reads the color of a specified pixel of an image), and noticed that the input coordinates are in Pixels, not in QC coordinates. Totally useless for using mouse coordinates to pick colors. argh.

can't you convert pixel coord to QC coord via the Rendering Destination patch ?

there's a sample patch in the dev folder that can sample color out of mouse coord.

Yep, you can convert. I've managed it as well with a few simple match patches. It's just annoying to have to do the conversion manually when nothing else uses pixel coords.

I was also confused because of the (just researched/posted) render-in-image imagePixel bug, which returns junk seemingly at random -- I was using a render in image to see if my calculations were correct, and they never seemed to get any more sane, which started driving me crazy :)

..use Pixel coordinates too.

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

wow, didn't notice that before. Good call :) I was thinking that it was like GLSL - normalized coords for 2D images, pixels for RECTs. Guess not.

.. since I've been using GLSL, I've taken to normalising coordinates in CIFilters anyway. Makes some things easier, I find.

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

I've no idea what you're trying with render in image, but I've had a few problems with it. Putting a clear as the bottom layer inside the render in image seems to fix most of it, so give that a go if you haven't already.

what bug are you talking about ? I'm heavily using RenderInImage --> Image Pixels (from within an iterator) and it is working great..... no bugs so far the only issue is the speed (as mentioned in the Image to Datastructure thread)

Here's a composition that demonstrates the bug. The image pixel patch returns wrong values for some regions of the time without the image accumulator.

To demonstrate it, trace the inner circle with the mouse. It returns black at points incorrectly.

[riiip = render in image/image pixel]

Definitely :) render in image w/o a clear is pretty risky :) On the other hand, you can get some cool undefined feedback behaviour if you do it just right :)

in your Render in Image macro, change the clear color to plain black / alpha 100% then values will be reported correctly - except for the Red component ( 32bit color / disable color correction into the RII macro)

One of the first compositions I made used the same accumulation bug in a different way. I did not know about the accumulation patch at the time and so I thought if each frame was cleared slightly I could get an effect like that. This was also in Tiger and controls of particle systems were not as good so it is kind of rigged inefficiently to get the effect I wanted. The disadvantage to this bug is when you get a slow frame rate or really fast motion of the particles you can see each instance of the particle.

I set those, no change on my machine. As a totally speculative aside, changing the clear color to opaque should have no effect since the rendered sprites would overwrite that with their own opacity anyway.

I tried fiddling with a few different formats, toggling a the RII check boxes, to no avail too.

Note the black sphere near the top of the inner circle. That's not getting any of the channels right but alpha, but alpha's 1 everywhere now, so that's difficult to screw up ;)

Preview	Attachment	Size
	riiip2.jpg	50.55 KB

Note: The above composition listens to your internal mic so as far as I know it will not run correctly in Safari.

I like this compo :) It takes a while on my machine, but the output looks really cool :)

To address the fast motion/slow framerate stuff, you could perhaps try using the line patch (new in leopard, or snag it from GL Tools on Tiger) to connect points? Not sure if it's worth the time to add all the necessary stuff for that though. shrugs

[oops, just kidding about the taking a while bit -- my mic was nearly muted, so there wasn't any input to drive it.]

well, i spoke too fast. I do have values, but extremely close to zero - but not plain zero - , except for Red

see attachment (btw, how do you do to show the pic directly on the page ?)

Preview	Attachment	Size
	Picture 1.png	112.28 KB

0.003 is about 1/256... so it's reading a raw value of 1 (out of 256) :) interesting.. . I have no idea what's going on with that.

As for pic on page, that's an admin-only feature I think. Easy to be abused, but I'll talk to smokris about it (there are a few format extensions that need to be added too, for the 3d files, etc).

On the image pixel patch, select 'rendering destination' as the colourspace. It's probably using some odd internal format otherwise, and it's the colours shown on screen that you're after anyway.

This is nice and clean, I like it. It's quiet here too, but having fireworks when you tap on the desk rules =)

I like that fix; less patches to clutter the editor. Between this or the image accumulator, it's simple to work around. Doesn't mean the default operation isn't a bug though :)

Thanks for finding this! Added to my report to Apple.

It was my screensaver for a long time because when there is activity in the room it provides a visual, but when no one is around or I go to sleep it goes dark and saves the screen. Thanks for all the positive feedback!

So, I was investigating the Sampler2D displacement crash bug more today, and I realized that there's one particular type of input image that doesn't crash the shader. This image type is output by a few patches like flame, plasma image, and some others.

This got me thinking: if we can transfer an input image of any type into this output type, we can work around the bug for the time being (with a performance hit, of course). Then I started thinking some more: OpenCV's CV->QC image patch generates these kinds of images too! This means if we convert a QC Image into a CV image, and then back to a QC Image, it'll be properly formatted, and we can safely feed it to the shader for displacement.

There are some caveats: the images must be square, apparently powers of two? I haven't fully explored the set of working inputs, since it crashes often if you get it wrong still (not sure if this is a CV bug or a GLSL bug).

Here's the setup:

Hopefully this is enough to get y'all started with useful displacement shaders before apple fixes the problem the right way.

Preview	Attachment	Size
	cvGlslHackaround.jpg	115.72 KB

I'll try this as soon as I get a chance! Great stuff!

PS Did you get my last post re. Multipass GLSL?

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

It's pretty cool, actually :) I got your multpass post. thinking :) e-mail me if you want to discuss things more in-depth.

(your post was actually what caused me to start re-investigating this problem, so that I could write a workaround sometime. ;)

[oh yeah: confirmed that texture needs to be square power-of-two (64, 128, 256, 512, 1024, etc.]

Just got me thinking... is it maybe the case that any texture can be used, if it's square and power-of-two? I'd never thought to try this before (though I tried all sorts of other things).

EDIT: Ah, no, sadly not... If only it were so simple.....

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

It ACTUALLY WORKS!!!!

That's very very very very very cool! Would it be easy to knock up a little plugin that just did this? Even better, do you think it would be possible to knock up one using the official Apple QC API (so I can roll it in to my compositions)?

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

Without knowing much about the official api, I'd imagine it'd be fairly easy (development name: Image Sanetizer, play on the word "sanitize", which means "make clean or sterile" and "sane" as in "rational, well-behaved").

But why can't you bundle in unofficial-API plugins? they're largely identical, as far as application support is concerned.

Oh, so you can 'roll-in' Kineme plugins by saving them with the Composition too? I'd assumed you'd have to distribute them separately with the QTZ, and tell people how to install them etc.

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

I'm not sure what exactly you mean by "roll in" -- compositions don't store plugins, but you can include them in zips or whatever. In typical application settings, the plugins go in Application.App/Contents/Resources/PlugIns/ or somewhere similar, and then you toss in a couple lines of code to load QC plugins from that path as well.

If you can store official-api plugins in a composition, please show me how this is done: it's a Really Bad Idea to have things work that way (arbitrary code execution exploit, anyone?)

..and choose 'Save Flattened Copy As....' from the File menu. Flattens any 3rd-party plugins from the official QC plugin folder into the composition (I think). That's why I was talking about the official API.

I'm currently writing QC files for use in a VJ application called VDMX, that can use QC Comps with specifically-named published inputs as realtime effects, or play them like other video material, but with access to all the control splitters.

It's still in beta, but is shaping-up to be a great app for realtime video!

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

I'm not sure what flattening does, but it's not bundling in plugins like you suggest: make a simple compositions with some (official api) plugins scattered in, save the flattened version, delete the plugins, and try to load the composition:

Preview	Attachment	Size
	flattenedError.jpg	45.92 KB

...saving files that have virtual patches in them. check the release notes for qc leopard for more info.

virtual patches is something i've been longing for so much, but the implementation doesn't really address why i want them. c'est la vie.

toby

http://machinesdontcare.wordpress.com/2008/03/26/radial-vertex-displacem...

This implementation isn't quite there yet, but it's an intriguing first attempt..

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

Just to clarify a bit on your post:

The only problem is, it runs quite slowly on my machine. It’s undoubtedly much faster to do this kind of thing with Vertex Buffer objects, but that means custom OpenGL code in a QC plugin, which is way beyond my capabilities currently. Sadly.

It's slow in part because of the presently slow hack-around (it's doing lots of redundant work as currently instituted, it could be cut approximately in half or even lower). VBOs won't help much, since that's already how it's working -- the glsl grid patch generates a fixed vbo/va grid that lives on the gpu after being uploaded, so it's the vertex shader that does the work of displacing them into interesting positions each frame - the displacement is a bit expensive, but assuming it's the gpu that's doing the displacement (not the case for intel gpus, but for all other ones it's likely the case), there's not much more that can be done to accelerate it. The cpu shouldn't be handling any vertex data at that point (making it a bit more tricky to optimize :)

One way to test whether it's the displacement or the image conversion would be to remove the conversion step (i.e. feed it a static image, or plasma image or something that animates, but doesn't trigger the bug) and see if the framerate goes up. if it does, it's "conversion limited", meaning the image conversion step is the bottle neck. If the framerate's the same, then it's likely "transform limited", meaning the vertex displacement stuff is the limiting factor -- see above paragraph on that.

toby, you never cease to make me look like an idiot for not knowing anything about QC :) thanks a bunch for your insight into the more user-centered features (us devs never get close to users, that's dangerous ;)

Thanks for the clarification there. There's definitely a way of doing this kind of thing directly in OpenGL though, and making it at least work on GPUs that don't have Vertex Shader sampler2D support. I'm thinking of something like the HeightMap QC plugin in the Developer examples folder. A version of this that worked on radial coordinates, and generated texture coordinates (and ideally normals too) would be verrry cool. Don't know if it would actually run faster though.

I'll do those tests you suggest and get back to you with the results.

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

I forgot about the whole "render to vertex array" technique for heightfields (that might also use vbos or something?) -- I never quite got the hang of that (because I didn't need to, and it's a bit more complicated than I was willing to deal with at the time).

I've never seen anyone use that technique to do non-planar deformations (or, generations(?) since you're generating the vertex data with the gpu), but that could be an interesting area to pursue (I'm sure it's possible).

It would be interesting to see the performance differences between those two techniques (as well as the geometry differences). I look forward to seeing that :)

Ah.. I was confusing VBAs and VBOs.. easy mistake to make if you're stoopid (like me).

I'm having fun with this VDM stuff!

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

http://developer.apple.com/samplecode/PBORenderToVertexArray/index.html

Works on hardware that doesn't support GLSL Vertex Shader samplers. It's a bit complicated though. Dunno if this is the method used in the HeighField plugin.

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

Yep, PBO R2VA is exactly what the GLHeightfield plugin uses.

I tinkered with that plugin, as well as the sample you linked to, but they were too complex for what I was needing to accomplish (What I needed at the time was a hardware-accelerated mesh blender, which took several weighted displacement vertex meshes, and produced a single mesh with the appropriate weighting. PBO stuff isn't really suited for that purpose, while glsl vertex shaders are).

Thanks for digging up that link :) I was going to later today, but you saved me some keystrokes :)

Glad to be of service :D I'd be intrigued to know how complicated you think it would be to add texture support to the HeightField plugin. It's a bit useless at the moment...

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

So, back to the crashy sampler2d in vertex shaders. I think I see exactly what causes it now: linearRGB.

I was checking out the image debug info patch, to see if I could figure it out, and I noticed that patches that work have genericRGB as the format. Everything else seems to have linearRGB.

As a quick test, I wrote a CI filter that does nothing but sample the input pixel and return it unchanged. The input is genericRGB, the output is linearRGB, and it causes an instant crash.

So, the question is, how do we convert back to genericRGB cheaply, and what is the difference between the two?

The output of any CI filter will crash it, regardless of what it actually does (you can return a constant color, for all it cares - no texture look ups or anything).

What I've found is that it's an image provider problem more than a colorspace problem (or, at least the only images that work at all are of a specific provider). Enable "QCShowExtensiveToolTips" in the option+Preferences (System) menu to see that info. Only QCBufferImageProviders seem to work. I can try to do some testing with different colorspaces within this provider to confirm whether or not colorspace plays a part in this issue.

Technically speaking, the difference between the two is one of colorcorrection (the bane of my existence). One of those two doesn't do any correction, while the other does (I think generic is uncorrected, while linear is? I'm not positive on that though).

"conversion" between the two may be as simple as changing some flags. Or, creating an image in a different colorspace that's derived from the parent one (the CV Hackaround does this in an indirect way).

I had no idea option preferences existed! Do other applications have a special preferences when you hold option?

Is it possible to set the colorspace within a CI filter? I could really use the sampler2d in my next project, and custom patches will be impossible for that (it's going to be a really dirty hack, i'm using an exported .mov file with embedded audio and vector art (embedding vector art seems to only be possible via a hex editor ;) )

I'm pretty sure it's colourspace and not provider related btw. I just did a test:

1. use a flame image patch, connect it to the displacement map input, it works fine. Colorspace is generic RGB (which is straight colour, no adjustment).
2. Now plug the flame image into an image crop, and send the cropped image to the GLSL patch. Provider is QC_Cropped_Image_Provider, Colorspace is still genericRGB.
3. Finally, plug video input or something into the image crop. The provider is still the cropper, but the colorspace is now ICC based RGB (or linearRGB, depending on what you plug in). It now crashes.

dwskau:

There's quite a few places where option - click works. One really useful one is option-click to launch an application from the dock - some apps will open an options box. I use that to switch between itunes libraries on the internal and external disks.

Once you noted the colorspace stuff, I thought it interesting (certainly worthy of investigation), but internally I also thought it was just a coincidental characteristic that you'd uncovered.

Your tests, however, made it much more compelling, so I did some tests myself.

Results: You're right, it's definitely colorspace related.

The test: I took out one line of code in the CV->QC image patch that disables color correction (miraculously added in by me for performance reasons, of all things!), and lo, it crashes (as your assessment would have predicted). Fascinating. Good sleuthing too, by the way. So, in essence, the image needs to be in the GenericRGB colorspace, and color correction needs to be disabled. uncorrected linearrgb doesn't work. (The Image With String patch generates images in GenericGray, which works too -- interesting because creating grayscale images in QC usually doesn't work, and it's not an rgb colorspace)

Notable Notes: With render in image, you can disable color correction, but the colorspace stays LinearRGB instead of the necessary GenericRGB. So there's no way to work around this using existing patches that I can think of (render in image and image accumulator being the only patches that do colorspace conversion as far as I can tell) -- and they're both hardcoded to output LinearRGB colorspace images.

As for setting colorspace in the filter, unfortunately I don't think that's possible (with all the options enabled, there's still no control over how the output image is allocated and configured, which is where colorspace magic takes place).

GLSL displace with movie : To make it works you need a quicktime movie with no compression (4/4/4), and enable "disable color correction". Video size must be a power of two (ex : 512x512 pixels) A movie loader output with this "nocodec" is GenericRVB.

As movie with no compression are 30 to 40 Mo/s with SD, you can just render your displacement map in one channel (as in the video noise exemple) and keep others channels black...you will then have a video with 12/18 Mo/s.

yanomano.

I've been thinking more about the GLSL/colorspace thing. I may be wrong here, but I think the difference between GenericRGB and LinearRGB is just the way the colors are handled - i.e. there is no difference in the data.

If that's the case, then just telling the GLSL patch that the incoming data is GenericRGB might fix the bug. I've been trying to do this today via the javascript patch, but unfortunately my javascript is well out of date, and I've forgotten most of it :/

Anyway, if you use Log(inputimage); you get a nice bit of text in the log containing the full image resource description. I've compared Generic and Linear inputs, and the difference seems to be nothing more than the native colorspace label and an extra bit in properties marked 'disableColorMatching = 1;'.

All I need to do now is to modify those two parts, and pray. Any idea how? The input image seems to be an object, but doesn't seem to behave like one. And I've not found any info on 'reverse engineering' an object too.

a good lead. cwright's method with the OpenCV image conversion does work, but I'm sure it's doing loads of stuff it doesn't need to, in this context.

Good work guys!

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net