Shader + Mesh + Sub patches = Skanky Consumer Patch?

you need to drag the plug to your /library/graphics/patches folder for it to show up in qc. the plugins folder is for official qc plugs not skanky. if the patch still doesn't show up make sure you have a description xml file in your project. also make sure you have declared qcplugin as the principle class in your plist file.

Thanks for the reply, but I havent made myself clear i suspect. I've got all that sorted. returning 2,0 and 3 within that function works and I can do things like "HelloWorld" and play with strings (see the Helloworld example) but I cant seem to make a consumer patch as there is something im missing. The plist is the same as the one from helloworld.

Im not changing contexts or doing anything crazy. I was looking at Apple's official SDK on this and i've pretty much copied their example to draw a simple Quad (the helloworld of OpenGL) but how does one create a consumer patch (the blue ones with layer numbers)

You'd think that wouldnt you? If thats all there is to it then sadly, something else is wrong. I have already said i'd tried this.

You'd think that wouldnt you? If thats all there is to it then sadly, something else is wrong. I have already said i'd tried this.

No. You cannot pass an "FBO" to another patch, but you pass textures (so, color attachments in FBO parlance).

If you use the public API, this is all possible. Im not 100% familiar with the skank SDK, but in my examining and experimenting with it a few times, it struck me that image handling for outputting a texture (say rendered via an FBO) seemed to do very odd things, leak, etc. PIxel Buffers and CPU side images seemed to work ok. This may have changed with recent SDK updates, but I also know the docs on that aspect were kind of odd and unclear.

You declare a consumer patch via:

+ (QCPlugInExecutionMode) executionMode
{
   return kQCPlugInExecutionModeProvider;
   // Alternatively you can use:
   // kQCPlugInExecutionModeProcessor
   // kQCPlugInExecutionModeConsumer 
}

you might also want to add

+ (int)timeModeWithIdentifier:(id)fp8
{
   return 1;
}

why don't you upload some code instead of saying my plugin doesn't work. the only thing people can do is guess as to why your code doesn't work.

heres my apple script plugin it is a skanky consumer. this might help you i don't know.

can you explain this...

Quote:
(I drag the plugin into my project but nothing appears)

the plugin is showing up in your patch list and you drag it to the editor but nothing is there even when you zoom out ?

the only place you can drag a plugin in from is the patch library. if your trying to drag in the plugin from your project that is wrong and will not work.

Because my code currently matches the helloworld example with just an extra few lines to draw a quad. Its not the code that is the problem. It's the concept.

The Library Inspector (I believe) is where one can see the list of all the patches. Dragging the patch from there into your project does work for all patches EXCEPT when that function returns 1. Clearly something else needs to happen or my idea of writing a consumer patch is wrong.

Yes that is what I saw. kQCPlugInExecutionModeProvider is just an enum though and a similar variable exists within the Skanky SDK in QCPlugin.h. I had hoped that returning it would sort it out. The client I'm working for wishes for subpatches and therefore, I believe, Skanky SDK is the correct choice. Perhaps I need to override this function as well? I shall give it a try.

Ah, passing textures is great. that should work fine so long as it stays on the GPU :) Cheers Vade

Thanks for that :) . Im looking at the patch and I notice you are not linking against the skankySDK framework in that example. Thats probably why it works for you. Again, im not sure exactly where this SDK fits (and wouldnt use it if the client hadn't asked me to). Nevertheless I shall soldier on.

Keep in mind you wouldn't be able to sell anything that uses it in the apple app store, and stuff based on the skanky sdk could stop working with updates, with no warning. It's likely you can get the pixels from a scene, do stuff, and output image, while not creating something rii based.

...going back to my cave.

It would need to be an image provider macro type, or just a non- rii based type plugin that reads pixels, maybe.

There aren't any code examples of an RII type of patch floating around.

Also...checkout the headers.

You'll get a better idea of what types of macros there are depending on OS (Render in Image was different between Leopard and SL).

I guess you'll have to get the depth image to do some of the effects you're talking about.

Note that there's an RII patch already, and there have been copious examples of DOF/tiltshift/etc. I guess you never know when it could be nice to have built in... I'll try not to think about it too hard, peace :)

Yeah that code is from the older sdk. I haven't tried the new one out yet other than looked at the examples so things may be different. Honestly have never tried sub patching so I can't offer any help in those regards. You would think returning 1 is the way to go as that is what worked for me in the past. When I get to my computer I will try to return 1 and look at the qc logs try and see what is going on.

In the .m file you have to specify:

+ (BOOL)allowsSubpatchesWithIdentifier:(id)fp8
{
   return YES;
}

...to allow for subpatches.

To make an RII patch, there are a number of other issues.

In Leopard, you had to specify execution mode:

kQCPatchExecutionModeRII1050

In SL it's:

kQCPatchExecutionModeRII1060

...so I don't know if QC will reconcile that difference for automatically if you just use kQCPatchExecutionModeRII1050, but I would think that one would have to program the patch to execute one way or the other depending on version (not 100% sure about that, but I would think it could be an issue). I wouldn't count on just being able to use one or the other, and having it work great in both circumstances... especially since Leopard didn't have kQCPatchExecutionModeRII1060.

Looking at the header for QCRenderInImage gives some hints about what you would have to deal with (but not about the execution mode stuff).

There's some stuff in there that looks a little out of the norm, compared to other patches. For instance, it doesn't seem to be using the :

+ (BOOL)allowsSubpatchesWithIdentifier:(id)fp8

...to allow subpatches within it. Instead, it looks like there are some executeSubpatches methods.

Thanks guys! Whatever it was in that example, it worked. There must have been a setting or line I had missed. At the moment, my patch is a consumer that simply executes subpatches within itself. What Ive been asked to do is apply shaders to content rendered using the Kineme3D patch. After playing with that, my patch executes a bunch of subpatches that load some test geometry. I believe there is a lighting example that would probably be a good starting point for adding basic lighting to an otherwise normal scene.

that would be wicked a new lighting patch... the current lighting patch in qc sometimes doesn't work as expected. like if you displace some vert's in a shader the lighting patch doesn't know where to render the shadows etc... even a patch with some of the skeleton tracking stuff your doing in cinder would make a cool plugin. there are not that many people making plugins. particularly with the skanky sdk. i'm not really an open gl guy or at least on the lower level. i like using it in qtz and open frameworks etc... glad you got it working.

There aren't any lighting macro patches that anyone has programmed for QC that I'm aware of, as the built in Lighting patch pretty much exposes all possible options besides flat shading, but if you have a macro already programmed that's "doing something", you can enable or disable lighting using normal openGL code.

Keep in mind that you can already apply shaders to content rendered with Kineme3D. I don't know your level of familiarity with QC, but consider a few things -

There is already a GLSL environment that allows you to manipulate vertices of objects and pixel shade the exterior of the objects.

Then, there is also the ability to place things within a Render In Image patch, and just pixelshade by using Core Image and render to Billboard or Sprite (or whatever), as well as the ability to Render In Image to a GLSL program with a Quad inside of it.

One of those routes could probably be involved. If you can get the actual depth image of the current scene, you can use that as part of a setup to do blurring based on that mask info.

The "shadow" thing is part of the Lighting patch, but it's sort of not part of the Lighting patch proper, as far as OpenGL function goes.

All of the openGL functions that the lighting patch exposes work fine, and as expected. The lopped on shadow/shader thing that is present 10.6+ is problematic, and unfortunate.

I havent had a proper look at the existing shadows code. I've been tasked to write 3 shaders that will no doubt use several FBOs and what not

An issue i'm now having is that the subpatches are executing but the actual macro patch isn't. I've been playing with some of the existing code for say, doing something simple like messing with the matrices and then executing the subpatches but it seems not to be working at all.

It seems that simply starting with a template doesnt work and that simply copying and changing an existing patch does (or at least I thought it did). Not impressed so far.

Also why is this occuring?

23/03/2011 11:46:43 Quartz Composer[5358] *** Patch class "Kineme3DObjectLoader" is not allowed in safe mode

I have allowed the mode within KinemeCore Preferences for ALL applications. This should not be occuring.

Ok so here is the code I'm playing with. Its just a mashup of the Field of View example from the GLTools examples I found. sadly, it doesnt work at all :( Conceptually, I think I'm missing something here.

Ok so I found this bit is different in the Plugin / Principal class:

KIRegisterPatch(ShaderTest);

replace with:

if( [manager isNodeRegisteredWithName: NSStringFromClass([ShaderTest class])] == FALSE ) [manager registerNodeWithClass:[ShaderTest class]];

Not sure what is going on here. I would suggest someone add a "Consumer Macro Patch example" to the existing XCodeTemplate download because I'd have never have spotted this if it wasnt for Dust

The only up to date published code I'm aware of that has an environment that executes subpatches within it is the Texture Transform patch I made a bit ago.... and I don't think I put any kind of licensing with my copyright on it, so if you use it, please let me know (I'm not trying to stop anyone from using it or benefitting, just curious when/where it gets used or mutilated).

I think that the point and line structure projects I put together are also the only current renderers that work with the skanky sdk and don't throw exceptions, so those may be helpful as well, even though they aren't macro's. They're both in my Dropbox at my webpage, and probably knocking around here as well.

Anyway, point being, that project will show a macro with subpatches executing, and with the macro manipulating what is inside as well.

Some old GL Tools published code is extremely similar and probably worth looking at (it's available if you trudge through past GL Tools releases... last published source was around Leopard), but hasn't been updated to work with the latest skanky sdk. So, if you compile, you'll probably see some type mismatches, and there are some places where old GL Tools doesn't set stuff up in the most up to date way, or maybe just doesn't work (in one or two cases).

Starting with a template definitely works though, so just double check whatever you're doing (you've uploaded no code to scrutinize). Double check what you're doing in your principle file (eg., are you including the patch), that you're updating your plist, an that you're doing the right thing with your xml.

Take some more time to understand the system before being impressed or not impressed. Once you have your patch, you can link your scene graph together modularly and manipulate it on the fly in realtime - this is SUPER DUPER lacking in Cinder. Neither is better/worse, it's just a different paradigm.

Do you understand that if you wish to output image/do fbo stuff, you can't use a normal macro, you have to use an RII type macro? The image has to render to something.

Here are projects for a structure point and line renderer that are 100% correct with the new sdk:

http://www.box.net/shared/3r3cvk5lio

... and this is a consumer macro that's 100% correct with the new sdk:

http://www.box.net/shared/x53rzjcbn0

Maybe use QC some before diving into whatever you're doing. It really sounds like you're marginally familiar with it, and getting pissed about developing with something that isn't even a documented SDK, that's being provided by a second-hand party. If I didn't understand how QC worked yet, I wouldn't dive into making a plugin using a largely undocumented spi.

For example, I wouldn't accept a project to work with Cinder, even though I'm largely familiar with it, have used it since the first release, and have built projects from scratch... and that's a documented SDK.

If you couldn't have figured enabling consumer mode without whatever Dust said (sounds like it's still not working though), take a few steps backwards, because this could really blow up on you, and your client. It seems like you still need to understand the various patch types (eg, macro or non consumer macro, rii type, provider, processor, consumer renderer). I apologize in advance, because there's no easy/soft way to say that, and I can't seem to keep myself from writing it.

Anyway, I do want to help, so feel free to scrutinize the links above and use however you need.

...before I EVEN dig into this more...

You eliminated the skanky framework and piled in a bunch of the QC headers.

Back up, look at examples.

Importing the Skanky sdk, like in the samples, takes care of all you need to do as far as the QC headers go. Eliminate that stuff from your project as a first step.

Look at the skanky sdk header, and note what it's taking care of for you... all of that stuff.

You're calling your principal class SoftShadowsPlugin in your plist, but it's SoftShadowsPrincipal.... I would recommend keeping naming consistent, and also making sure principles are Classes, and your patches are Patches (eg., pay attention to the hierarchy).

There's probably some more stuff going on with this one, but fixing that stuff is going to be a start.

Things move fast. That code was out of date soon after I psoted it. I've gone with the helloworld example and added the modifications to make it consumer. It appears this imports the framework as a framework and seems to work so far.

Im having issues with the FBO being properly bound (apparently it wont bind inside QC but is fine elsewhere). There is probably something else not immediately obvious I'll need to track down.

Thanks for the examples. I'll look over these and see if there are any differences with what I have.

Fbo's can work in qc.

I'm just guessing, but I think the impasse is that you need to understand that in qc, this would normally mean that you make a macro environment that also has an image output, not a consumer macro.

Look at the difference between something like the built in fog patch and the render in image patch, for example.

Hey don't feel bad. I had to ask how to set a string with the leopard skanky sdk. I wouldn't hope for to much though in regards to examples. As it is now there are three of them where as before the skanky sdk was just an xcode template without examples so you sort of had to explore the headers your self. It was a hack enabling peple to be able to write a plug for qc when it was not possible. Or at least that's what I have read as that was before my time. Even though it's an undocumented framework there are clear benefits to programming plugins on the dark side.

Essentially, adding lighting and shadows to a mesh would, I think be taken care of by using kineme3d (its mesh and mesh renderer patches) WITHIN my lighting patch. This would mean that the lighting patch would end up as a consumer as it contains a consumer?

In the case of basic OpenGL lighting, lets say, I would assume that my simple GL lighting patch would just turn on and setup lights, execute its child patches, then turn off lights. That seems natural and would seem to fit what I've read here. This doesn't work. In fact any operations I perform in this manner dont seem to work but this is in direct contradiction to a thread I have read here.

http://kineme.net/Discussion/ProgrammingQuartzComposerPatches/HowtocodeM...

This is extremely frustrating as this appears to be quite a simple and elegant way of doings things, if it worked. I suspect it must do but there is something else I've missed. I have gone over the code, checked the XML, the plists, naming conventions and indeed, rebuilt it several times from scratch. I've looked at the ports and built a future FBO class (as I'll need that for the shaders for shadow mapping). None of it seems to function quite right yet :S

Also, yes, I notice the RII stuff now, looking at the QCPatch header. Not sure what that is doing but it certainly is doing things differently (Ie black screens of death! ;) )

Quote:
Essentially, adding lighting and shadows to a mesh would, I think be taken care of by using kineme3d (its mesh and mesh renderer patches) WITHIN my lighting patch. This would mean that the lighting patch would end up as a consumer as it contains a consumer?

I think there's something muddy here.

OpenGL doesn't support shadows directly in any way. You could use the stencil buffer or depth buffer in OpenGL, or something else, but it's not some "boom, here's shadows".

I think you likely know this, just need to figure out how this can work for you inside of QC. You haven't really gone much into how you're trying to achieve shadows ... I think you also mentioned something about DOF, which is a whole other ball of wax, that also has multiple approaches.

If you take an environmental macro using the Skanky SDK, or for that matter, make a layer type consumer that doesn't render anything using the stock SDK, it really is going to boil down to something like, Enable GL Lighting... then you're going to specify your light source, and whatever the other minimal parameters needed are.

For instance, if you were making a "Cube" or "Sphere" patch from scratch (eg., not an environment), you could build in lighting, like in a normal GL program. This is the same kind of thing you'll do in programming a macro environment, you just won't be any geometry that you're rendering "in" the code of the patch. You'll need to keep track of current and last light position, etc., and follow all typical rules.

Let me repeat though... the stock Lighting patch pretty much exposes all built in GL Lighting function. The only issue with it might be when you reduce specular (or something like that) to some super small value. I can't quite remember, but it's quite an obscure issue. I guess it doesn't expose cone lighting/GL_SPOT_CUTOFF, but I haven't really thought too much about that until this second.

When I think about doing DOF, blurring a scene, doing a shadow algorithm in QC, my mind keeps thinking towards having everything in something that renders the scene to an image. Then, you would do whatever the processing is, based on that (say, if you made a render in image type patch that derives depth, or does some kind of accumulation buffer stuff), and render that to a Billboard or something else after making doing the processing to composite your shadows (or whatever) with the main scene.

http://http.developer.nvidia.com/GPUGems3/gpugems3_ch19.html

I don't really think there's any kind of lingering great facility to have a normal consumer macro, and fullscale change the appearance of the entire scene to be "blurred" or "have shadows" without rendering, and then layering on top. I would guess that the shadow algorithm for QC must be reading the scene, doing some shader hyjinx, and laying it on top of the scene after some post processing...but that's very involved, has a whole other non-QC related framework involved in it, and sucks.

You could probably place your subpatches scene in a macro that isn't a consumer, and is RII based, get your necessary info, output images, do your processing, and wind up with a great looking scene. There's probably not even a need for a special patch to achieve that stuff (at least I don't think so), though it might somehow make worflow easier.

I'm still slightly muddy on what you're trying to achieve, and the algorithm you're trying to pull off. It's hard to say if it's possible with a normal consumer macro or not (I tend to think not).

Wooooooooah woooah! You are getting ahead of yourself here.

Ignore all shadows and DOF. I've written all of these and know the score. That is not the issue. The issue is, as has been said, the concept of taking these algorithms into QC and knowing the architecture and the APIs given the requirements handed down to me.

I mention shadows and lighting and DOF as these are the end results. These are done but they need to be married in with QC. But that isn't really a concern at this point

What is a concern is that things I expect to work are not given my incomplete understanding of the SDK. So far, Ive spent two days and have acheived very little. I seem to get small little titbits of code here and there, like the thread i've seen and im trying to put together a picture of the workings. So far, its been a slow process.

What is an RII patch? 10 minutes of googling has not resulted in an answer. Is this relevant? I dont know? Given my understanding of a patch, why would this simple snippet not work? By not work, I mean the Quad is not drawn. That previous thread implies that OpenGL commands can be executed and this would tally with the basic examples that Apple provides, but of course, this is a QCPatch, not an official SDK one. Certainly, I'm pretty confused.

- (BOOL)renderQuad:(double)time arguments:(NSDictionary*)args {
 
   glMatrixMode(GL_MODELVIEW);
   glPushMatrix();   
   //glRotatef(90.0,0.0,1.0,0.0);
 
   glColor3f(1.0, 0, 0);
   glBegin(GL_QUADS);
   glVertex3f(0,0,0);
   glVertex3f(1.0,0,0);
   glVertex3f(1.0,1.0,0);
   glVertex3f(0,1.0,0);
 
   glEnd();
   glPopMatrix();
 
   return YES;
}
 
 
-(BOOL)execute:(QCOpenGLContext*)context time:(double)time arguments:(NSDictionary*)arguments {
 
   // Execute Subpatches
 
   [self executeSubpatches:time arguments:arguments];
 
   CGLContextObj cgl_ctx = [context CGLContextObj];
   if(cgl_ctx == NULL)
      return NO;
 
   [self renderQuad:time arguments:arguments];
 
   [mBoolPort setBooleanValue:TRUE];
   [mOutputImage setImageValue: [mInputImage imageValue] ];
 
   return YES;
}

Im trying to test my understanding of the API by using small examples, (such as turning on or off OpenGL Lighting, or drawing a quad) in order to see if what I perceive the SDK to be is indeed correct.

It's kind of aberrant to have an environment macro that also renders geometry... but it would basically be like this (reference the Texture Transform link I sent earlier):

- (BOOL)execute:(QCOpenGLContext *)context time:(double)time arguments:(NSDictionary *)arguments
{
   //   double modifiedMatrix[16],t1,t2,t3;
   GLint oldMatrixMode;
 
   if([inputBypass booleanValue])
   {
      [self executeSubpatches:time arguments:arguments];
      return YES;
   }
 
   CGLContextObj cgl_ctx = [context CGLContextObj];
 
//Then here is where you would write some geometry inside of a consumer macro, even though it's super duper weird / not normal in QC.
 
glBegin(GL_QUADS);
   glColor4f(WHATEVER POINTS);
   glVertex3f(WHATEVER VERTICES BLAH);
   etc, etc.
   glEnd();
 
//Now we're back to doing the function that the macro environment stuff actually does, we're done rendering that piece of geometry.
 
   glGetIntegerv(GL_MATRIX_MODE,&oldMatrixMode);
 
   glMatrixMode(GL_TEXTURE);
   glPushMatrix();
   glLoadIdentity();
   glTranslatef([inputXTranslation doubleValue],[inputYTranslation    doubleValue], 0.);
   // Rotate the matrix
        glRotatef([inputXRotation doubleValue], 1., 0., 0.);
   glRotatef([inputYRotation doubleValue], 0., 1., 0.);
   glRotatef([inputZRotation doubleValue], 0., 0., 1.);
 
   glScalef((1./[inputXScale doubleValue]), (1./[inputYScale doubleValue]), 1.);
 
 
   [self executeSubpatches:time arguments:arguments];
 
   glMatrixMode(GL_TEXTURE);
   glPopMatrix();   
   glMatrixMode(oldMatrixMode);
 
 
   return YES;
}

Soooo, I'm not saying this does anything functional, just that you can make a macro that has subpatches with that render geometry as well as performs a function of some sort, and also have that macro environment render some kind of geometry. I just happened to have this handy, and it's a macro type project.

RII means a "render in image" patch

Thanks for that. It confirms what I thought. Essentially this means one could bind and unbind shaders, turn on and off lights etc and have these state effects cascade downwards.

Of course, I've dropped in your code and it doesnt work for me. Therefore I reckon I've got the concept but something else is missing. At least I know I'm on the right track. Cheers

Ah, that is indeed useful and makes a lot of sense now. Essentially similar to deferred shading. there certainly is a lot of terminology one needs to get a handle on. Cheers

Ah, that is indeed useful and makes a lot of sense now. Essentially similar to deferred shading. there certainly is a lot of terminology one needs to get a handle on. Cheers

Render In Image is in no way similar to deferred shading. Deferred shading is a multiple step rendering technique that moves lighting to a secondary (or later) rendering pass to easily allow additional per pixel lights, shadows etc at the cost of needing to draw and pre-render out multiple g-buffers for normals, albedo, depth, color, etc.

Render in image is basically and FBO with a depth + color attachment.

Deferred shading might be able to be implemented with some decent use of the render in image patch, but it would be wholly inefficient. You really need to support multiple render targets and shaders that write to multiple attachments at once, which the RII patch does not.

Deferred shading is a technique, RII is a specific implementation of Rendering to a single off screen buffer.

In the past, I've Rendered to a set of FBOs and then recombined to do SSAO, shadow mapping etc in a deferred shading manner. However I get what you are saying with the RII patch now. That would be a bit inefficient if you had to pass the results as an image into another patch and then another etc then combine said images. Even if the images are actually textures I guess that would be quite bad.

RII may not be relevant just yet though. Ive got some FBO code written anyway but there is something else within the macro patch i've written that is not working. The consumer i have executes its subpatches but does not affect the OpenGL state in anyway. Apparently, given the examples, it should.

I would have thought this was how the GLSL patch would work. Essentially binding a shader, executing subpatches and then unbinding the shader before returning YES, all within this macro consumer patch. From what I've seen from the texture example provided, this would indeed be the case (although not shaders, matrices in that case but hey). If im not going mad then I suspect Im missing something else I just need to track down.

I'll keep hunting, see what comes along.

The issue with RII is you would need to duplicate your scene contents and load the scene and render it once for every g-buffer you need, whereas with MRT and shaders you can render out via one FBO and one go all g-buffers.

But clearly you get that :)

As for OpenGL State not cascading, understand that depending on what sub-macro objects you insert into the chain, they are also in essence black boxes and may re-set state, over-write state, change loaded GLSL shaders, active textures and active texture units, etc, all behind your back. QC is middle-ware, and you cannot make assumptions of what state is when you get it, and if you change it, you need to change it back to its previous state, not the default state.

Yeah, that is another thing. I've noticed lots of pushing and popping of the various states in order to sort that. I guess I'm back at square one here. At the moment, it seemed the most natural solution as the end result is to have a lit mesh with shadows. Fairly straight forward, you'd think. The v002 and Kineme3D bits seem to take care of meshes for me, which is great.

Now im looking at the lighting patch which (and Im guessing here) is doing shadow mapping and lighting with a shader of somekind. I know this sounds like reinventing the wheel (and will indeed check this) but I guessing that this patch works how I've described. Sure there is potential for state changes but thats true of any potential system really. There could indeed be things going on within a patch that you dont want a macro patch to mess with. that I get, but since the lighting patch appears to do this sort of thing, is my thinking off the mark or not? I suspect it isnt but since I cant get my patch to do simple state changes over static geometry, i couldnt get that confirmed.

@vade: Implementing an RII/fbo type patch from scratch could support multiple render targets.

Quote:
Now im looking at the lighting patch which (and Im guessing here) is doing shadow mapping and lighting with a shader of somekind. I know this sounds like reinventing the wheel (and will indeed check this) but I guessing that this patch works how I've described.

The built in shadow option, that has been lopped onto the Lighting patch, is sort of a giant hack. Before 10.6, this shadow option, and related parameters were not present.

As it has been implemented, it has some issues: -It can look like crap, if it isn't tweaked well. -It appears to look like it uses a really low res texture, and there is no exposure to us, as users, to tweak this. -There is no exposure to the smoothing algorithm or shader without major hacking, to substitute our own. -It breaks the QC iterator (eg., if you use the QC iterator patch with it, all iterations dissapear). -It's lopped in with the lighting patch. -It can be tricked into placing shadows in the wrong places or texturing the wrong sides of objects.

Also: -It's part of a private framework called "MeshKit". Which makes it super easy to access/deal with.

For all of that, it can be made to look pretty sweet if you play to it's strengths, and then placing your scene in an RII and doing some rounds of pixel shading (like your DOF or Tiltshift type stuff), can make high quality looking scenes.

It appears that QC somehow sends a structure of image data that the shaders in meshkit process, and then lay on top of the scene in QC. How exactly all this works is undisclosed, and private to Apple.

It's not even part of the QuartzComposer framework, so I don't even know exactly how the bug fixing process goes down on that one, or getting more info on how to replicate the exact way it works.

It looks like you might be able to get the depth buffer, do whatever you need to do, render it to a polygon, and then lay it on top of the scene using the correct blend mode, all within one big consumer macro.... that seems kind of weird, but maybe it might work. I'm doubtful about that though, because I don't know if one can render geometry "over" subpatches in the macro like that, while reading what's below. IDK.

The more typical thing would be to make some kind of Render in image patch that outputs the images you need, and do a kind of deferred rendering (arguments about definitions/semantics notwithstanding). I use typical super loosely, because no one has really bitten this one off exactly, only similar things.

Quote:
Sure there is potential for state changes but thats true of any potential system really. There could indeed be things going on within a patch that you dont want a macro patch to mess with. that I get, but since the lighting patch appears to do this sort of thing, is my thinking off the mark or not? I suspect it isnt but since I cant get my patch to do simple state changes over static geometry, i couldnt get that confirmed.

Again... take it with a grain of salt, but the shadow patch was a horrible hack, it doesn't work right, it breaks QC, and I don't know what the hell whoever wrote it was thinking. It was not the right way to attempt that, and probably shouldn't be used as a model for anything. That sounds horribly negative... I just know you're a talented guy, and it sort of sucks to see you surveying the land, so to speak, and wind up getting caught up in the black hole of the QC4 shadow function.

I would definitely think about emulating some kind of deferred system, or attempting to get the depth image, "doing stuff" and then rendering that to 2D geometry.

Or geez, this would be an endeavor, but you could always render all of your geometry, lighting, etc., inside of one big patch/program, and have total control (I'm writing that one off as crazy talk...)

Sadly, I havent even gotten to that point. The code you sent before doesn't work for me and I simply cant get my consumer patch to do anything except execute its children. It wont even publish any inputs or outputs so something else is very wrong. I need to sort that out before I get down into the shadow depths.

I actually have a stand alone app i've been toying with that does deferred rendering of a loaded Quartz Composer comp using FBO+MRT. I've not spent much time, and have not added post processing, shadows, DOF or anything like that yet, but it is doable if you want to treat QC as an asset rather than the environment to do the shading in.

Need to add MSAA+FBO blit in there for AA, but it ought to be doable. Part of the issue is materials, textures etc for diffuse rendering need to respect the shader implementations, and may not output normals properly, etc.

Preview	Attachment	Size
	Screen shot 2011-03-24 at 12.19.45 PM.png	104.13 KB

Of course. I never said it couldn't ;)

Ok guys. Just wanted to say thanks for all your help! I couldn't have ironed out the lil bugs without you. There is a lot of very subtle things going on here that I'm not sure about but Im now playing with Shaders and FBOs just like I would normally.

I shall write all this up and once I've shipped it, I'll make the source available and then no-one else may tear their hair out! :)

Cheers!

So what is it you made?

Ah, well I've hit one final snag. executesubpatches only works once. I believe this is to do with the internal graph structure that QC takes care of. I suspect it only evaluates each patch once or as many times as it feels it has to. I believe there is an "Iterator patch" or something similar that I need to look at? does this ring any bells?

Essentially, the basic shadowmapping algorithm means I need to render once from the light, once from the camera. So really, I need two executions of the subpatches. Is there an invalidation flag or similar? I notice the

• (NSArray *)consumerSubpatches;

And have tried iterating myself but I suspect Im missing something (again!)

No wait, I think I have it! :P

Looks like you can execute patches again but for some reason, their parameters arent updated .

NSArray *subpatches = [patch subpatches];
   NSEnumerator *e = [subpatches objectEnumerator];
   id object;
 
   while (object = [e nextObject]) {
      QCPatch *p = (QCPatch*)object;
      [p setNeedsExecution];
      [p _activate];
      [p execute:context time:time arguments:arguments];
 
      [self recallPatches:p context:context time:time arguments:arguments];
   }

This appears to affect the trackball patch. I have, within my custom macropatch a trackball and within that, a set of teapots. I can change the teapots and see the effects but the trackball does not appear to update its values. I suspect my hack isn't quite right.

I made this!

:D

:D

THANKS GUYS!

I can now eat this month! :P

Preview	Attachment	Size
	shadows2.jpg	57.23 KB

Nice. Congratulations on getting it working.

Small comment, it looks like your soft shadow falloff is inverted, no?

How do you handle texturing in your soft shadows implementation, as QC can switch between sampler2D and sampler2DRects behind your back (or per user settings, or 3rd party plugins, etc)?

Will this be available to play with?

Yes I noticed that falloff in that shot. Its quite different to the first implementation. I'd started off with Texture Rectangles but that was proving to be a little more messy than I'd like as there were things that I was unsure about with QC that were confusing me.

There is an issue with face culling that, unfortunately, can't really be controlled in each pass. The cubes and teapots can all have their culling settings set by the user overriding the shadow map settings so I'm not sure what best to do with that.

Actually, there are two issues here. The first is you can see the shadow on the front face, repeated on the back face as there is no lighting calculation step included in the fragment, though that can be fixed. Also, the blurring isnt quite right. I think, with VSM shadows there is no attenuation or falloff and the effect you describe is the penumbra getting squashed up close to the shadow casting polygon.

Annoyingly, I suspect VSM is probably not the way to go here.

What filtering approach are you using to soften the aliasing? What's your texture size?

If you control culling when generating your texture, I think you can get rid of some of some of the wrong self shadowing problems ( I think you sort of referred to this above ).

Have you made a test macro to see if you can over-ride culling setting on an object that's in it yet? If you disable front faces, it works and over-rides object inside, as long as the user didn't enable culling on back faces ( I spent about all of 3 minutes checking). You could just have the caveat of telling people to leave culling alone on their object. There might be a way to mitigate that... didn't check.

So, to be clear, in a test I did:

glCullFace(GL_FRONT);
glEnable(GL_CULL_FACE);

... along with the other boilerplate stuff.

I popped a sphere in the test environment, attached a texture to check, and front face was culled. So, you do have a chance of manipulating that so you can cull front faces when you sample to get rid of some of that self shadowing.

I don't know if this is better than what Apple's doing though (eg., the steps you're going through/the implementation). What are you trying to work around, as far as the Apple shadow implementation goes? You can get the edges fairly soft, and can use alpha channel to control the intensity as well.

The Apple implementation does break QC's iterator, so if yours works and doesn't break that, it's a winner in that regard for sure :)

"You can get the edges fairly soft"

you can't. The built in lighting patch is lame, has a poor resolution, bad looking and slow.

It has a setting that jitters the texture, and makes it softer, and you can control this. You can also add manipulate alpha, to get it not as heavy looking.

I'm not arguing about criticisms, only that many shadowing techniques can be made to look bad with bad settings, and that if one was to implement a similar technique with a higher res image than what Apple's using, it would likely be slower. If it had less respect for geometry in the scene than Apple's it would tend to look worse (shadows rendering on stuff they aren't supposed to).

Preview	Attachment	Size
	shadows.jpg	91.2 KB

Im liking the shadows on the image you posted. That is pretty much bang on what I need. I was asked to implement something like this:

http://developer.download.nvidia.com/shaderlibrary/docs/shadow_PCSS.pdf

Thing is, PCF is allegedly not that good and I'd already started with VSM. I suspect you are right and that shadows are really tricky to get right and depend on a lot of variables which one really needs to pay attention to. At the moment, my default shadow map, adapted from the cinder example works well but is very sharp. PCSS choses a different kernel size given the distance from any occulders which makes sense.

I spoke with the client and they are really not keen on using the built in one. Fair enough. They pay the bills :P

I need to look over VSM again because I think it will work with a 'good' gaussian blur filter (which is a real bottleneck at the moment). It doesnt shade correctly but there is no lighting component specified yet which i need to look into.

Yes, with the culling, I will have to say that the culling cant be touched by patches within the shadow patch. Thats just par for the QC course I guess. I think I do break the iterator though. I have to draw geometry twice and that means a hack where deliberately call the patch execute methods recursively :S

Preview	Attachment	Size
	shadow.jpg	42.46 KB
	shadow3.jpg	40.77 KB

Actually, looking even more into it, Summed Area VSM is the best technique but it looks tricky! And also Vade, I think you are right, there is an odd inverse of the falloff with the shadows. I cant figure out why but it "appears" to be part of the VSM algorithm itself.

So far so....almost good?

http://www.vimeo.com/21891223

So far so....almost good?

http://www.vimeo.com/21891223