|
|
OpenCL Depth Map / Texture Map Kernel (Composition by gtoledo3)
This composition demonstrates an OpenCL kernel that reads a Depth Map and Texture Map. It then positions vertices based on the depth map, and outputs color structure based on the texture map. When used with the Mesh Creator, this setup creates a textured mesh with normals that can be used with the QC4 shadow engine, or be subjected to further mesh processing. Please read the notes inside the qtz. Please make sure that your depth image and color image are exactly the same pixel width and height. Swapping images on the fly while the Viewer is rendering is possible if both images are the same width and height, but not generally recommended. If you try to edit the kernel or setup and get a freeze, or any kind of exception, please restart your computer just in case. This is rendering one vertex for every pixel, and does so in a very quick way, but it's not a good thing to interrupt the chain on the fly, given the nature of the kernel. It will act stably for you so long as depth and color image are identical w/h and one leaves the image inputs to the kernel undisturbed once you initialize the comp. |
This works very nicely with the video input associated to both or either image input.
The result, with an iSight and in built iSight is spookily reminiscent of a "warts & all" self portrait I achieved , after several hours worth of work & experimentation , some several years ago.
Today a couple of minutes [& all your coding time].
Very smooth work is my initial impression.
Cool. Worked well here, after a little fixing (without POT textures I got a badly corrupt output mesh.. not sure if it wants POT or just multiples of 16). It nearly works with mismatched image sizes too, although your texture coordinates seem locked to the depth map's image size so the mesh was ok but the texturing was off.
Looks good though. I'm going to have to get into opencl at some point. My new work mac supports it in actual hardware which will help :)
Hmm, that's interesting, there aren't any POT embargoes on my GPU apparently, with this kernel at least.
I've tested this with a few builds. On some, it's not a problem at all to have different sizes between the color map and the depth. On others, it freaks out. That's why I made a big deal of noting to "be careful".
Yeah, I locked the texture coords to the depth map, because I didn't think it made any sense to expose controls for that on the kernel; I figured anyone could use a image transform or crop, etc. if they wanted to offset one from the other; it seemed like depth texture and color texture should be the same size. Do you have any other thoughts on that? I kind of figured that someone could size the depth map up or color map up with a resize if it was lower/higher res or something. Hmm.
You should have a really good running start for sure with OpenCL. I'm sort of surprised at some of the things I haven't used it for, that are starting to dawn on me.
I'm going to have to post some more pics. I had that color and depth map handy. This can look pretty darn hi-fi since the vertex count is uncapped (which can also freak your computer out if it can't handle it).
Not sure on the depth map / colour map size thing. I suspect there will be cases where it's better to have them different. A few possibilities:
Maybe the xbox kinnect has different res colour/depth outputs? Maybe somebody here can confirm :)
Maybe you want a kinnect providing z, and a nice HD cam doing colour :D
Sometimes a lower-res depth map is good for performance. A lot of stuff I've done looked good at 128x128 for depth, and 1k x 1k for colour. I was combining a depth map with a normal map - you get high speed plus detailed looking 3d. This might not apply the way you're doing it, and you could handle it differently in CL.
Interesting that I get texture problems that you don't get too - I'm using the software CL layer which in theory should be less stringent. Weird!
And yeah, I should investigate CL properly. I never really got the chance so far - it plain didn't work when it came out as I have an older GPU here, and recently I've been doing lots of iphone stuff rather than QC. Speaking of which, time to see if I can figure out openGL profiling in shark :/
Couldn't resist having a poke at that openCL code. It's now running in 8bit mode ;) (This would be a ton of fun with a kinnect hooked up!)
Hey, that's a cool one... I probably would have just thought to run the source through a block filter, but why not do the stepping when it's so easy? Cool idea.
I started doing some openCL centroid calc based on the premise, that works pretty darn well in x/y/z.
One thing I started noting, is that if a pic already has depth blur, or if you're sampling from movies where stuff is generally moving, it's not too crazy to get a kind of pseudo depth map going. I've looked at some interesting techniques of taking an image with some form of depth/motion blur to it, adding controlled blur, deriving the dif, doing an edge detection on the dif, and then using a laplacian process with some tweaking to get a usable depth map. Some versions of this technique use two consecutive frames for analysis. Some versions do a little touch up pass, where depth map is manually touched up. There are some interesting variations on the techniques of "3d from one image), and they're way more usable than I thought... with several caveats of course.
Yeah, there are cases where you can do 1-cam depth extraction extremely well. Best case would be a static scene with plenty of contrast and colour variation where the camera pans round so you get lots of parallax - if you do a straight motion estimation, the motion speed IS the depth map.
There's way too many cases where none of the methods work though, so unless you're planning to use this where you're in total control of the content and can design it to work well with your algorithm, you still need some kind of depth capturing camera/lasers/kinnect if it's indoors.
The cube look: actually this is a really efficient way of doing it. Even if you have it set to render just 1 cube, it's still rendering vertices for every input pixel. I'm sure it's possible to output actual cubes instead of just moving the vertices around, I need to learn CL a bit more to figure that out though :(
Any good resources for CL? Or if not, how do we use the work size inputs?
Oh yeah. That stuff could only possibly work when you're in total control, and as an "editing after the fact" scenario, so that you could mix depth map info that you've derived from scenes with motion with tweaked depth maps for times when motion isn't getting it. I'm not really talking about realtime at all at this point (though the handdrawn depth map stuff is a good way to get some kind of "3D" out of static 2d backgrounds).
You can render a cube, sphere, ripple, whatever, along the lines you're talking about. Of course, since you have all of the vert info, you can potentially render anything at any vert as well (aside from what's going on with the mesh creation).
If you read the last paragraph on the OpenCL kernel descrip, does that explain what's going on with worksize for you?
Here's some good stuff:
http://www.khronos.org/files/opencl-quick-reference-card.pdf (a quick ref sheet)
http://developer.download.nvidia.com/OpenCL/NVIDIA_OpenCL_JumpStart_Guid...
http://developer.apple.com/library/mac/#documentation/Performance/Concep...
http://www.nvidia.com/content/cudazone/CUDABrowser/downloads/papers/NVID...
http://www.cmsoft.com.br/index.php?option=com_content&view=category&layo...
http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf
In the ADC ref samples, there are some QC related kernels/implementations. That's worth a download as well. There are also some OpenCL Xcode projects.
I think right now, I'm personally pretty interested in converting some GLSL fractal type routines to OpenCL.
Some really good links there, GT.
BTW - to whom do we refer Apple Dev Site peculiarities to?
I speak here of the amusingly cyclical & recursive results of seeking to go to http://developer.apple.com/library/mac/navigation/#section=Topics&topic=... - which is a standard link upon the sidebar on the Apple Developer site which results in the following repeated state
Well, you could find the site map, and try to figure out who to talk to, and send a note. Conversely, you might just go ahead and write your note on a bottle and fling it into the ocean. Either way. ;)
Nice collection of links there :) And I can't believe I totally neglected to check the patch description in QC. Not that it helped actually. I've managed to get it to crash plenty, but not actually do what I need.. just noticed that you can check the work size values when they're set to automatic though, and they're not what I was expecting! I'll try again at lunch time.
Sooo.
This is an update that adds function that turns and color vectors under a certain length alpha, and/or any over a certain length alpha. When in over mode, this effectively culls your front or back geometry.
In summary "floor chops the back, wall chops the front, make sure it's not in Replace blend, probably use Over."
I'm uploading it in context of a setup for the kinect.
Ah...OpenCL.... So much potential, so many crashes... Still.
The whole Work Size issue is one of the things I've always struggled to get my head around with OpenCL. It's all-important though, because of the lack of failsafe memory-management on the GPU. I'm sure this explains a lot of the OpenCL-related crashes, too.
What I found reading OpenCL documentation was that an innordinate amount of time was spend explaining the intricacies of the memory/resource-management system, and very little on the actual language itself, beyond it being basically 'C-like'. Unfortunately, since you don't get to tweak any of this backend stuff in QC, the vast majority of the OpenCL information out there is pretty useless in a QC context, in my experience. My suspicion is that OCL is potentially very fast, but how fast it actually is able to run in a real-world context depends to a very large extent on the intricacies of parallelisation and memory-usage setups which QC doesn't expose, for understandably reasons.
On most of the occasions I've been impressed by how fast OpenCL kernels have operated, I've found oldschool GLSL vertex shaders have been able to get the job done much more efficiently. This obviously doesn't apply to all scenarios, since OpenCL is theoretically much less limited than GLSL, but I'd say don't assume OpenCL is always the fastest way of dealing with, particularly, 3D geometry, if it's also possibly to do the same thing in GLSL.
a|x
You should try serious a/b testing on the fps thing you're talking about. Extrusion shaders can be really quick and stable for sure, but this gets quicker fps than GLSL does for me using a grid of comparable resolution (of which, isn't available in QC stock). In addition, the macro itself is doing a number of functions GLSL isn't doing - giving me vertex and color outputs, as well as normals. My fps with a comparable GLSL shader that is actually doing less, is less.
When I run a vertex distortion with GLSL as opposed to OpenCL, it's quicker with OpenCL.
It might be beneficial to revisit the whole fps comparison thing sometime. I guess I'm kind of going "hmmm" about the "much more" part of your comment, as I've made many setups in GLSL and replicated them in CL, and vice versa. It's darn even steven, and I've usually been disappointed with GLSL once I have it operating at the same resolution. I'm sure you were seeing a dif, I'm just curious what/why.
Extrusion is probably a rare case - in GLSL we have to use a vertex sampler, which tends to be not well supported and probably quite slow from what I've seen. If we could create a mesh from the texture the using GL buffers, it'd probably be much faster, likely faster than CL.
Yes, you're right about Vertex Texture Fetch- it's certainly not universally supported. When it works, though, it seems to be much faster than doing an equivalent thing in OpenCL. Take the example of the line-displacement effect in the clip on this page: http://www.dazeddigital.com/music/article/9045/1/weirdcore-visuals
I did this first as a series of OpenCL kernels, one processing each line, and was getting much slower framerates than when I simply ran a single GLSL shader, fetching the displacement texture in the VS.
a|x
Well, I tend to do a bunch of performance testing.
Extrusion is generally quick on my GPU using GLSL, nVidia 9600. It's faster than doing the same thing in a comparable plugin.
Yeah, but didn't you say you were iterating for some reason? Is it really a true a/b? (Not saying it wasn't.) It seems like that can be done in one kernel. I'm not sure comparing a string of kernels to one shader is analogous. I think you can create alpha areas like that using sin functions, and do it in one kernel.
BTW, I <3 that look. I have a shader that is extremely similar working in GLSL because of a project I was asked about, but never got a call back on.
cool got the depth with wall patch working. i have been messing with some of these depth field displacement patches. i had to drag myself through the process kicking and screaming as for what ever reason when doing these height field kernel tests my computer crashed after crash.
i have got into the habit of stopping the player when coding. but sometimes if i just move the player window i will get a crash. weird might be kinect issue ?
note a resize which can and will crash your kernel if you have not defined your workgroups.
i found its best to actually define your function __globals size and to not let qc auto magically crash.
so i have found actually kineme 3d height field seems to produce the best realistic texture. my cl kernel texture looks nice as its almost the same as your patch gt.. but there is some flattening in spots that should be extruded where your latest patch has a clearer and more defined depth reconstruction but the texture is knarly.
don't want to hi jack your thread here but we should trade patches and see how to get the best definition with best texture. if it interests you i will post.
been meaning to set up a website again..
Using them as is, or with some kind of extra editing/manipulation/playing around?
Yeah, an OpenCL kernel is directly accessing your GPU, so all bets are off unless someone does something to sanitize that. It is a really bad idea to edit the kernel or make connections on the fly. I stressed that in the meta notes in the composition. This is designed to take images of equal height and width.
There is definitely nothing automagical about it.
If you don't define global dimensions/size/output, it is assumed you are dealing with arrays of the same size. So, if that's the case, it's of benefit speedwise (on my GPU... I haven't tested this extensively), to actually let this happen automatically. Again, array size being equal on ins and outs is implicit. So, two images should be the same size, or all bets are off. A crash or something whackadoo would totally be expected.
So, you can use that as your basic deciding factor on which route to take.
I've found it to be very flexible and stable, and also allow one to go fairly high rez, but I disagree based on these reasons:
-K3D heightfield creates aberrant "noise" on edges that is not accurate. This is obvious at higher levels of extrusion.
-K3D heightfield doesn't look as good with the stock GL Lighting, and doesn't render correctly with the Shadow option.
Meanwhile, OpenCL can render a mesh that works perfectly with lighting, and that delivers a vertex for every pixel, does a number of other calculations, and still gives my 60fps on the 9600, and has ability for robust post processing, and gathering of data about vertex and color positions. Sometimes I'm shocked, because my computer seems like it's literally doing nothing when running very complex OpenCL kernels once the chain is finally tuned, and I know GLSL and K3D methods of vertex manipulation fairly well, and the limits I do or don't run into.
I'm not knocking GLSL or K3D routes as they both have their strengths. I just think some of the "crash and burn" stuff people get with OpenCL is very easily explained, and not inherently fault of the tech.
I'm not 100% sure what you mean. That's chunky points, so the texure is going to be gnarly. It's a bunch of squares! It's not a photo of triangle mesh. It's also a demonstration of the raw kernel and functions, not so much a demonstration of how to get it ultra-sweet.
It looks like you're filtering your depth channel with a blur. The depth image, when accurate, obviously gets a bit of a stairstep to it, because it is sampling every centimeter (or whatever...looks finer probably). If you render that without GL Lighting, it's going to result in a nice texture where you can't see any ridges. Note attached picture with <30 seconds tweaking. I've had some nice results using shaders on the resulting mesh to smooth ridges, and also have a function I'm working on in the actual OpenCL that is seeming to do a nice job at it, but bears some tweaking, so I don't unleash anything dumb.
...but as I was saying, I'm not ultra convinced about using blur to interpolate depth steps. It's not really the same. It is awesome for a blob effect. v002 blur driven into 4~5 range looks like an excellent silly putty man thing. I'll have to look through the things I've been prepping the depth channel with, and relay them back, because I've found a few doozies, for inaccurate fun looks.
I would say, post your patch! I haven't seen anyone else make an OpenCL kernel that displaces by one texture, and outputs color based on another, besides the one I posted. It's not a rocket science premise or anything.
I have a tendency to get so involved in my own projects ... I have never taken a look at the boids stuff yet, other than opening them up and hitting play on all of them! I was so burnt on it after my "node based boids" experiments.
I have my own personal hit list on this that I'm knocking off one by one. I would say, post your stuff, if it's awesome we learn something, if it's not awesome we learn something... that goes for my stuff as well, not just meaning yours ;) I'm reallllly concentrating on my own list of things though, so don't take offense if I'm slightly laggy on chiming in.
Let me ask... when I tried building that oFX QC blob thing you posted in one of the first kinect threads, it totally failed. Any thoughts on that? I'm curious to see it.
I wrote an explanation on the alpha color assignment for vertices depending on luminance and threshold/wall input values, to effectively let someone cull geometry from the front or back without degrading the input image on my blog, www.georgetoledo.com , for anyone interested in a breakdown. Since OpenCL is new-ish, it took a couple stabs at figuring out a workable approach, and then getting acceptable syntax.
I ended up creating a kind of funny routine around this... I'm taking the verts, normals, and colors, and processing them in a kernel that's multiplying the amount of coordinates, and making hairs:
http://www.vimeo.com/17697427
that's nice to hear George. there are to many good things to come from posting similar work, then to argue about who did what first, so i'm glad your looking past this.
so I was trying to do this before much in the same method as you by extruding brightness with a line, and cl would go down frequently. it was very off putting. I started with a dev extrude example sort was getting ok results. got some interesting results with blur and k3d height. defiantly a Ginger bread man if you don't fill in the alpha but then with to much blur steps it would seem like I was under water.
I decided to take my own advice I mentioned and define my work groups and global output sizes. once I did this I was able to have a few hour session programming while in execution. couldn't believe it not one crash even when I was mangling some code. so I'm not trusting qc to automatically define my work group and sizes any more. as I have never had a cl session without going down at least a few times. so I think I'm on to something now.
yeah that qc blob thing was just a plugin I made from the from the fox kinect source. it will no longer run as kineme has released the kinect tools patch which uses the same library so it renders my temp plugin no good. that's why it crashes. you can however remove the kinect tools if you want to try and run it. sort of obsolete now as the ofx kinect add on has really matured since I made that plug.
Sigh.
I'm out.
Here's the thing.
If you examine your statement, there is implicit insult. It's not technical critique.
that was not my intent brother as i was very explicit with my statement. i'm sorry you read more into it than was my intentions. i don't really see how me saying "i'm glad you looked past us posting similar things this time" as an insult.
although i guess given recent history i can see how you could read an implicit insult out of my statement maybe ?
i don't understand as that was not my intentions. cant we just look toward the future forget the past, wipe the slate clean, start over or something, call a truce. i don't care if your right and i'm wrong or your wrong and i'm right. i just want to share some patches.
im an honest person, i truly was happy and thought you looked past us posted similar things again. you your self said we can both learn from each other vise versa. i'm sorry if i was strait forward and expressed my happiness in these regards. my statement was literal, i don't know if i spoke to soon but i understand given our history how you could take it the wrong way and read something implicit.
cant there be room for people to post similar patches ? i feel it serves a greater purpose as there are always many approaches to different problems.
This is a thread on forum, about a given topic; a composition/depth computation kernel I posted, and some related compositions.
Things that seem to be implied as acceptable conversation:
-"There is something technically wrong with this, and here is what I think it is."
-"This is crashing on my system. My system is ___."
-"I like/dislike this stylistically, because ___."
-"I recognize this code as being MIT, copyrighted, etc., by I or someone else, and this is the substantiation." This is technical critique, imo.
-"I use this route instead, because ____."
Things that seem unacceptable:
-"I think you're saying/doing X because (insert personal, non technical reason)." This is why email and private messaging were invented.
-Excessive one on one back and forths that do not pertain to the group, or that are off topic, or especially, personal in nature.
-Unsubstantiated hearsay.
-Slanderous character assassinations.
I think this is something that we all need to strongly consider when addressing our peers on this forum.
Let's get back OT! :)
good list george. i agree, we should keep things to a technical critique rather than anything personal. we should all be big enough to take a technical critique wether in negative or positive light as just critique and to not take offense to anything within those terms.
I suggest this list gets put in the wiki or somewhere high level where it get's noticed by all users on the new Kineme site. Good list.
-