vImage Erode filter

superflea's picture

While looking around for tips on creating an Erode filter (http://www.texturemaker.com/help/functions/filter_erode.htm) for learning purposes I found the source for it on the vImage documents on Apple's developer page (http://developer.apple.com/library/mac/#documentation/Performance/Concep...).

Thing is, in my ignorance, I have no idea where to stick the code. Is it supposed to be a CI filter? Can I even do it in QC? There seems to be no tutorial or guide in the document, any clues? Thanks.

gtoledo3's picture
Re: vImage Erode filter

vImage is one of those things that seemed awesome, and then went away. It's been superseded by just using CI, GLSL, QC, etc., as far as I know. I very recently asked someone about this, that I guess would know, and it was like "don't bother".

Vade has great examples of how to do an erosion filter, and an erosion plugin... that could probably be setup to do more passes. Somewhere around here there was a thread about image processing, and I posted a link to a great page that breaks down many fundamental algorithms.

I would think about implementing it as CI, CL, or GLSL, before using something like vImage.

vade's picture
Re: vImage Erode filter

vImage only works on the CPU side of things, and has highly optimized convolution, erode, dilate and image format transforms (as well as tiling and transformations). vImage is useful for when you really need to use algorithms that don't fit well on the GPU, or at least where you can't rely on certain GPU features across the board (say you need to support integrated graphics, for example).

vImage is useful for munging things for conversions to and from specific formats, say, for OpenCV, or other sorts of systems where you have to do things on the CPU side, and going between CPU<->GPU and back is not worth the effort, time, or latency (usually these transfers introduce a frame delay or more, due to their asynchronous nature)

In my Kinect plugin, I use vImage to do some conversions, for example. I had a request for a multi pass Erode and Dilate, look no further than the Noise Industries implementations in Core Image for that, if you need them asap.

gtoledo3's picture
Re: vImage Erode filter

Is that kinect plugin out yet?

superflea's picture
Re: vImage Erode filter

Thanks, I didn't know Vade had one already. As I said I understand nothing of vImage, so it was just a case of having some code to learn from. Anyway Erosion filters seem to be covered! thanks again

vade's picture
Re Kinect / v002 Open Kinect

Still in beta:

http://v002.info/downloads/beta/v002%20Open%20Kinect.plugin.zip

Infrared / Color image from kinect

Floating point depth image

Tilt control inputs

Accelerometer.

Right now, it has a weird threading issue in VDMX that I am trying to solve, but works well otherwise. I've taken some liberties with "cleaning" up the depth map output so its somewhat pre-corrected, but frankly it does need some more work in that department.

gtoledo3's picture
Re: Re Kinect / v002 Open Kinect

Oh, nice, thanks! (I feel like a broken record, but it would be truly awesome to see the source when/if you feel like it. Your published source are basically the best examples of standard api QCplugin coding). I've done a serious amount of work in investigating various "cleanup" algorithms... I guess some of that might be better external to the plugin, as other patches, but you never know.

Maybe it might make sense to have a setting to change bit depth output? That's beyond my knowledge at the moment. I know that when I get intensity images in QC, I wind up having to render in image, and use a billboard, so that I can run it to CI or processing plugins and have stuff be reliably sane. CL can seemingly take the intensity image.

vade's picture
Re: Re Kinect / v002 Open Kinect

I've had no issues feeding the intensity images into various things. Core Image outputs an RGB image from the 32 bit Intensity last I checked.

Source is going to be released when the plugin is ready to go.

gtoledo3's picture
Re: Re Kinect / v002 Open Kinect

It's weird... if I feed it into certain filters, things work ok. There are some filters and plugins (cimorphology, laplacian, ni thresh, some others), where the image output starts flipping horizontally back and forth, unless it's rendered in image first. Maybe it's a gpu compatibility thing... I don't know. In discussion with cwright, I believe it has been written it up to CI not handling intensity images sanely/correctly. I know I've seen the flip flop texture on multiple systems though.

I'll try to get some kind of error demo together... it seems like it's already a rdar though.

vade's picture
Re: Re Kinect / v002 Open Kinect

Oh, and as far as cleanup is concerned, last I heard, the 11 bit values range from 0 to 2047, where 2047 = "cannot determine the depth of the image", but realistically, in almost all cases values are output from ~ 500 to maybe 900 or 1000, nothing past 1000 and nothing less than 500 in raw depth values (If I have those numbers recalled correctly).

Have you had experience otherwise? This is kind of a hobby project and not a super pressing, so I'd rather get a nice, fast, correct output than something kind of half baked (which it is now). My goal is to output a normalized, 0 to 1 for sane values depth map, which is re-mapped spatially to make sense (the depth map is "compressed" almost logarithmically, as you know).

vade's picture
Re: Re Kinect / v002 Open Kinect

Could definitely be an odd GPU capabilities thing. Sadly, Intensity images are not renderable by FBOs (its an invalid color attachment target), so you can't output a proper Intensity image from the GPU, otherwise things could be written to take in, and spit out and intensity image.

QCs image pipeline is really rather odd, imo. I'm not doubting you, I just have had no issues with it (yet).

gtoledo3's picture
Re: Re Kinect / v002 Open Kinect

"Oh, and as far as cleanup is concerned, last I heard, the 11 bit values range from 0 to 2047, where 2047 = "cannot determine the depth of the image", but realistically, in almost all cases values are output from ~ 500 to maybe 900 or 1000, nothing past 1000 and nothing less than 500 in raw depth values (If I have those numbers recalled correctly)."

That sounds about right... I think it might be able to resolve deeper? I can check.

I know that it can get usable depth info (eg., it can be used to "do cool stuff") at around 30 feet, which is better than the spec. I think that with OpenNI, the FOV can be adjusted (uncertain), which may sort of convolute that.

Yes, the depth map is compressed almost logarithmically... it's funny that you say that! That's been my exact experience... it's almost comical.

So, I guess you're going to do some kind of histogram op or something? (excuse me if I'm off track). I think the idea of having a truly normalized, high res, depth image, is probably a good one. Especially since OpenCL kernel seems to be able to resolve the bit depth... so it would be A+ to get more steps out of the CL mesh.

One weird thing... the kinect actually outputs slowly moving vertical scanlines.... I think like 2 or 4? If you up the contrast on the image, it's sort of obvious. I've had communication with primesense about it, and it's expected.

gtoledo3's picture
Re: Re Kinect / v002 Open Kinect

Well, my understanding is that the Billboard/RII "sanely" decimates the image to whatever your system can handle, but yes, it's not an intensity image at that point. I flip RII settings to 32 bit, and hope for the best. It at least makes things unflakey.

I've seen that "sketch-out" on the nvidia 9400/9600 stuff, and then totally screw up on some ATI 4xxx series (can't remember the exact # offhand). I guess I should check it on an intel I have at my disposal as well.

vade's picture
Re: Re Kinect / v002 Open Kinect

I've not noticed those scanlines. Is that in the depth, or the color image? Funny.

I was not thinking of a histogram op, I think that might be too slow (speed is always a concern of mine), and I don't mind being... somewhat static or having to do manual calibration.

I might try to expose a depth min and depth max (so you can pin ranges in the depth map, prior or during normalization), so you can in essence clip out background /foreground you dont want. Hrm.

I am definitely keen on keeping a much accuracy in the depth map as possible (which is why I dont want to downsample to 8bit intensity or RGB 8).

Here is a screenshot of the new version of the Rutt (re-written from the ground up to use some better, more modern OpenGL techniques) that can handle 640x480 from the Kinect without batting an eye. On my 330m its been able to handle 1080p at around 25/30 fps (and thats 1920x1080 vertices being displaced, not just video input). Fun times, I think?

PreviewAttachmentSize
Screen shot 2011-02-16 at 5.16.09 PM.png
Screen shot 2011-02-16 at 5.16.09 PM.png678.46 KB

gtoledo3's picture
Re: Re Kinect / v002 Open Kinect

I still haven't had time to check out the plugin...but re: your comments, this is a CL kernel the thresholds out the background or foreground. On gpu's that support CL, I've had several of these working, and it's been great... I think I had 12 working with it being pretty much the same as running 1. This does 60fps on my nVidia9600, inputting two 640x480's. It can be horrible on non-supported gpu's. I'm obviously decimating image depth by rii'ing in this setup.

It's kinda funny... I was experimenting with modifying the Rutt to do the same thing last night (near/far thresholding). It's cool that you're thinking along the same lines. I've noted how different the Rutt code has been as it's developed... the last one and the 1st are pretty darn different.

The scanlines are in the depth image for sure (but may be present in the rgb?). It's definitely there, but harder to notice unless you contrast the image. It's not a software or driver thing apparently, but in the hardware.

PreviewAttachmentSize
Kinect_Read Depth w Floor & Wall_gt.qtz26.25 KB

gtoledo3's picture
Re: Re Kinect / v002 Open Kinect

Oh, suggestion: it would be cool if you got the image input for point sprite working/added.

This may be crazy, but it would sort of be cool to be able to have a separate input for normal map. You can composite the rgb and depth, greyscaling the rgb, and run that through a ci filter to make a normal map, and get nice looking mesh, that has the appearance of higher quality than available with just the depth map (though it is obviously then affected by lighting conditions/rgb... still sort of useful).

I've been doing that with glsl currently. I know you have that "high quality" normal setting though... it can still be cool to manipulate the normal image (I think).

That screen shot looks nice, btw.

dust's picture
Re: Re Kinect / v002 Open Kinect

tried running this beta. for some reason it doesn't show up in my plugins ;( any suggestions ?

vade's picture
Re: Re Kinect / v002 Open Kinect

You running 10.6.6 ? should work 32 and 64 bit, but I kind of threw up a build on to the server. This is why I dont share shit till its ready, ha.

vade's picture
Re: Re Kinect / v002 Open Kinect

The sprite input is for Noise Industries Synth plugin, and those are not supported for end users for the QC plugin, but maybe I can add something to the v002 Rutt Etra.

Part of the issue with point sprite inputs is they really need to be mip mapped, and patching normal images from QC (normally rect textures) are a) large, usually full SD size or larger, b) cant be mip mapped by virtue of being rect.

This means for every textured point sprite, it has to do an incredibly complicated texturing process to map the large texture on to each and every sprite, once for every point sprite. Since we are probably dealing with a decent number of point sprites.. you can see where this is going.

This is not fast. This is incredibly slow. There is also fill rate issues since particles (there are many of them), when textured, and alpha blended.

This is why I never released the Bokeh plugin, it just does not scale well.

dust's picture
Re: Re Kinect / v002 Open Kinect

ya 10.6.6 good idea not to share until things are tight. i got an ofxKinect plugin working in qc. seems to be working in 64bit mode. i did try this one in 32 but still didn't show up. your movie player loads up though freaking awesome.

vade's picture
Re: Re Kinect / v002 Open Kinect

I have no idea. Ive not had any issues at all. Maybe you need to remove the ofxKinect plugin you made via ofxQCplugin, since you have multiple lib-freenect libraries being loaded. I suspect that could be an issue.

gtoledo3's picture
Re: Re Kinect / v002 Open Kinect

I'm getting some messages in console (it doesn't show up for me 32 or 64 bit either):

2/16/11 10:14:13 PM Quartz Composer[4552] *** QCPlugIn: Cannot preflight plug-in at path "/Users/georgetoledo/Library/Graphics/Quartz Composer Plug-Ins/v002 Open Kinect.plugin" (Error Domain=NSCocoaErrorDomain Code=3587 UserInfo=0x10066cc20 "The bundle “v002 Open Kinect” couldn’t be loaded because it is damaged or missing necessary resources." (dlopen_preflight(/Users/georgetoledo/Library/Graphics/Quartz Composer Plug-Ins/v002 Open Kinect.plugin/Contents/MacOS/v002 Open Kinect): Library not loaded: /SVNRepositories/libfreenect/build/lib/libfreenect.0.0.dylib Referenced from: /Users/georgetoledo/Library/Graphics/Quartz Composer Plug-Ins/v002 Open Kinect.plugin/Contents/MacOS/v002 Open Kinect Reason: image not found))

Hope it doesn't wind up as a pain in the ass for you... thanks again for sharing, it looks like it was probably something simple.

I removed anything else I had that might have libfreenect installed (kinect tools, and another plugin). Still no luck.

vade's picture
Re: Re Kinect / v002 Open Kinect

Ah, Derp. I see what I did there. Pain in the ass it is. I need to include the Dylib and change the link path. Will deal with this tomorrow.

vade's picture
Re: Re Kinect / v002 Open Kinect

Try the same download link again. This has an embedded framework, and, uh, ought to work? :)

gtoledo3's picture
Re: Re Kinect / v002 Open Kinect

It's still complaining about libusb not being loaded over here; not showing up in the patch list.

gtoledo3's picture
Re: vImage Erode filter

I just noticed that the apple example histogram op plugin does use vImage as well.

vade's picture
Re: Re Kinect / v002 Open Kinect

twitch