v002 Open Kinect Beta

vade's picture

Hi.

I believe i've sorted out the linking issues for libfreenect and libusb, so the plugin ought to load.

32/64 bit intel on 10.6. Gives you floating point depth image output, tilt control (+- 30º), color or infra red image, and accelerometer output. Right now, the device input selection does not work, so its limited to 1 Kinect only.

http://v002.info/downloads/beta/v002%20Open%20Kinect.plugin.zip

I've taken some liberties with attempting to optimize the depth image output (normalized between 0 to 1, floating point intensity image), so it may not match what you are used to from the Kineme Kinect patch.

Let me know if it works, hopefully it will actually load. Performance should similar to the Kineme plugin, even though public API plugins have some issues. This is optimized to go direct to OpenGL (it outputs textures, not pixel buffers).

Thanks, feedback welcome, especially regarding the depth image. I would personally like very much to have the output color / infra red image pre-corrected for the depth map, and the depth map pre-normalized and linearized for straight rendering. If anyone has tips, algorithms etc, that would be great.

vade's picture
Re: v002 Open Kinect Beta

Oh, the only real issue I have noticed is that there is a subtle threading issue in VDMX B8 that needs sorting out. Otherwise I have had no issues in QC.

gtoledo3's picture
Re: v002 Open Kinect Beta

This is really loading for you? I can't figure it out! I'm going to check on another install... will be back...

vade's picture
Re: v002 Open Kinect Beta

Yea, I also had a friend try it before I posted it, loaded for him.

gtoledo3's picture
Re: v002 Open Kinect Beta

I just tried it on a totally clean install, with no OpenNI, no KinectTools, no anything non-standard, and it's still not popping up in the patch list.

I wonder if it's just me!

Is it possible that the framework is somehow loading for you, but not from the plugin? That seems crazy... but I'm trying to figure out how it's working on your system...or not on mine, whichever.

coebro's picture
Re: v002 Open Kinect Beta

Not comming up here either.

gtoledo3's picture
Re: v002 Open Kinect Beta

It's still giving me this kind of thing (didn't check to see if it's exactly the same as the past builds):

2/19/11 7:42:49 PM Quartz Composer[164] *** QCPlugIn: Cannot preflight plug-in at path "/Users/gtoledo3/Library/Graphics/Quartz Composer Plug-Ins/v002 Open Kinect.plugin" (Error Domain=NSCocoaErrorDomain Code=3587 UserInfo=0x168cd0 "The bundle “v002 Open Kinect” couldn’t be loaded because it is damaged or missing necessary resources." (dlopen_preflight(/Users/gtoledo3/Library/Graphics/Quartz Composer Plug-Ins/v002 Open Kinect.plugin/Contents/MacOS/v002 Open Kinect): Library not loaded: /usr/local/lib/libusb-1.0.0.dylib Referenced from: /Users/gtoledo3/Library/Graphics/Quartz Composer Plug-Ins/v002 Open Kinect.plugin/Contents/MacOS/../Frameworks/libfreenect.0.0.1.dylib Reason: image not found))

vade's picture
Re: v002 Open Kinect Beta

That asshole had libusb installed haha, so libfreenect was finding it. The last step was to tell libfreenct where to look, ought to be fixed now, I was able to re-create on another friends workstation, where its now loading. Goddamn.

gtoledo3's picture
Re: v002 Open Kinect Beta

Scorrrrrrre. I haven't tried it yet, but it populates the list at least. Somehow I knew it was picking up that framework from somewhere. Weird thing is, I have libusb installed too (maybe it's not the same version). So, I'm not going to try to think to hard about it, because I don't want my brain to hurt.

vade's picture
Re: v002 Open Kinect Beta

I had to run some post process scripts to change where the dynamic library loader looks for the .dylibs. I thought I had it, but forgot that libfreenect itself needs to know where libusb.dylib is.

Programming can be a real chore sometimes.

Relevant script call:

install_name_tool -change /usr/local/lib/libusb-1.0.0.dylib @loader_path/../Frameworks/libusb-1.0.0.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.plugin/Contents/MacOS/$PRODUCT_NAME" 
install_name_tool -change /SVNRepositories/libfreenect/build/lib/libfreenect.0.0.dylib @loader_path/../Frameworks/libfreenect.0.0.1.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.plugin/Contents/MacOS/$PRODUCT_NAME" 
install_name_tool -change /usr/local/lib/libusb-1.0.0.dylib @loader_path/../Frameworks/libusb-1.0.0.dylib "$TARGET_BUILD_DIR/$PRODUCT_NAME.plugin/Contents/Frameworks/libfreenect.0.0.1.dylib" 

Oh the things I do for fun. Very glad it is loading for you now, sorry for the pita, this shit is hard to test on a dev system, where getting it up and running at all is a pita.

coebro's picture
Re: v002 Open Kinect Beta

It looks amazing! Im sitting and having a very happy moment over the infared image. Thank you vade. Once again you make brilliant work.The tilt angle control also works great! I will get right to play with it.

vade's picture
Re: v002 Open Kinect Beta

Very glad it is seemingly working and outputting an image. Like I said, I did admittedly take some liberties with the depth image format. I would love feedback, because I have not done as much work with the Kinect as many of you have.

dust's picture
Re: v002 Open Kinect Beta

the depth image looks amazing.

i had issues with my libusb being in /opt/local/lib so i moved it to /usr/local/lib.

does this script let you load the dylb from the plugin ?

cause if does that would be pretty sweet... i'm trying it right now.

gtoledo3's picture
Re: v002 Open Kinect Beta

I guess it will take more testing, but it seems to not work w/ openCL kernel. GLSL is ok.

The throw of it feels kinda weird, but the non-linear nature of it may just feel exacerbated by the normalizing.

I think that getting the depth map so that luminance=near-z is a driver option, but idk...

vade's picture
Re: v002 Open Kinect Beta

For the opencl kernel you likely need to specify a different input buffer format, as a 32 bit intensity image is not a 8bit, four channel rgba image, no?

vade's picture
Re: v002 Open Kinect Beta

I would highly suggest not messing with that. The compiled dylibs I provide match the plugins expectations. Unlike frameworks if you provide an older or newer dylib things will break. Why do you want to do this?

vade's picture
Re: v002 Open Kinect Beta

Interesting regarding driver level flipping of the depth image. I can do it manually I guess if not. Also regarding throw, it's definitely not accurate right now, I've not done much correction or testing.

gtoledo3's picture
Re: v002 Open Kinect Beta

I see that on my gpu (or rather, in the QC/CL interface), it was a matter of colorspace that was messing with me, w/the cl kernel... it not only has to be set to 32-bit, but also Absolute Linear.

This works ok for me.

PreviewAttachmentSize
Kinect_Read Depth w Floor & Wall_gt_v002.qtz22.04 KB

dust's picture
Re: v002 Open Kinect Beta

just so other people could run the qtz file without having to post install anything i guess. the depth image looks nice. got make some new code to work with this.

vade's picture
Re: v002 Open Kinect Beta

No one has to do anything other than install the plugin. That was a post compile script to correct the libraries that I hvw included in the plugin, no one needs to do anything special at all.

That was more to illustrate how fucking stupid this kind of shit is. Hah.

vade's picture
Re: v002 Open Kinect Beta

Nice! I figured it was something silly, it should work, so I'm glad it does.

gtoledo3's picture
Re: v002 Open Kinect Beta

There are definitely more steps of resolution than I've been seeing. It's sort of weird to deal with the inverse image all of a sudden (again), but it's not a humongous deal. Don't take my comment about the invert in the driver as bible, but I think it's an option. I may be convoluting it in my memory with some other driver that's available. It's a boon to pure visualization of the depth data in qc though... it resolves more levels of depth, and that's really useful immediately, for visualization or utilitarian stuff.

I think there's something different about the infrared though... the rgb works ok without adjustment. Word to the wise for anyone trying it w/the cl kernel I posted, infrared into the rgb/texture input will make QC crash, at least over here. Seems cool otherwise. It might take some other kind of color setting, idk.

PreviewAttachmentSize
Kinect_Read Depth w Floor & Wall_gt_v002.png
Kinect_Read Depth w Floor & Wall_gt_v002.png331.55 KB

vade's picture
Re: v002 Open Kinect Beta

That's actually a good point. Right now when you toggle between infra red and rgb, the plugin outputs a different format image to save CPU-gpu bandwidth. Something to take heed of.

gtoledo3's picture
Re: v002 Open Kinect Beta

Ok, at least I'm not off my rocker. I hadn't checked image format yet, but I couldn't see any other reason for it crashing with the flip to infrared.

I'm really digging this, and seeing that extra depth resolution.

I'm posting a composition that has a couple of things going on :

It has separate inputs for depth map, texture map, and normal map. So, the normal map source should be a black and white image. Theoretically, it's for using a lower grid res, and keeping the appearance of higher quality normals. That works pretty well. You can also jack up the pixel offset to smooth/soften the mesh appearance, while keeping the actual vertex displacement and resolution the same.

The pixel offset is set negative to make up for the fact that the depth image is inverse. If one was to use a black and white of the color texture as a normal map source, you would swizzle the signs to positive to keep the lighting looking correct.

I haven't vetted this much, so there could be something screwed up...but I think it's ok. (It should be safer for messing around with flipping texture images, etc, than opencl, just no access to vert or color struct data.)

The infrared is really wild w/ normal map... I'm messing with that. Probably post something in a minute. Very cool.

gtoledo3's picture
Re: v002 Open Kinect Beta

I'm really enjoying this... this is setup for the infrared.

With this, b/c of the way the image is setup, when stuff gets too close to the camera, it pops all the way to the front... that isn't bad, it's just funny to think about/deal with.

This looks way hi-res compared to what I'm used to...the boon of the infrared and depth lining up automatically is nice.

It's too bad there's no way to get the color and the infrared (I don't think this is possible at all...not with your plugin, just in general. Not 100% sure.)

This is setup w/ two separate, and almost identical macros. One that's stock, and one w/ GL Tools polygon mode. So, if it doesn't restore b/c you don't have GL Tools installed, all you miss out on is a renderer setup with line modes and point sprite. There's still full texture renderer.

gtoledo3's picture
Re: v002 Open Kinect Beta

K, this has to be my last post with this for a bit... I got the right setting for the infrared texture with the OpenCL kernel. It's really slick looking.

gtoledo3's picture
Re: v002 Open Kinect Beta

Main question - How do you think culling back and near vertices (or making color alpha) is best achieved with a heightfield plugin like the rutt?

Are clip planes the right route? It seems like if that's happening in OpenGL, it's better than coloring a bunch of vertices alpha.

Just for background, I have my own private version of the Rutt that is just tweaked for my preferences... built in translation and rotation, blend mode and depth independent of wireframe modes, some extra blend modes.

BTW, in the context of the Rutt (sorry for not posting this at v002, but I'm already here, and it's semi related), I get warnings - for:

if(error = glGetError()) I get: using the result of an assignment without parenthesis. When I change it to what I think is right, I get the same thing, so I've left it alone.

Then:

if(error = glGetError()) NSLog(@"OpenGL error %04X", error);

... I get using the result of an assignment as a condition without parenthesis.

I don't get that one, especially. I use that log all the time... I think it's even just a default part of an Xcode QC consumer (?).

Maybe it's b/c in the recent rutt's you don't include Shader.h or .m.... so I've had to figure out what it would be.

In my .m file, I get the warning " conversion specifies type 'char *' but the argument has type 'GLcharARB const **' for:

:

  if( *theShaderCompiled == 0 )
  {
     NSLog(@">> Failed to compile shader %s\n", theShader);
  }

...and for :

if( *theProgramLinked == 0 ) { NSLog(@">> Failed to link program 0x%lx\n", (GLubyte *)&programObject); } /

...I get " conversion specifies type 'unsigned long' but the argument has type 'GLubyte *'

....are these things to worry about at all? I kinda hate errors in projects. Do these things just get converted (or typecast? Is that the right term?) automatically? I could do a shot by shot, but sometimes I find myself "fixing" this type of stuff, changing argument types to be correct (not necessarily just in context of this), and then find the build won't compile at all. I run the compilable code, check for errors, log messages... everything seems cool. So, it makes me miffed sometimes... it's a grey area to me right now.

Man, this image is really nice looking. I'm resolving facial detail that I just wasn't getting before, even in some non-QC projects.

vade's picture
Re: v002 Open Kinect Beta

if((error = glGetError()))

gtoledo3's picture
Re: v002 Open Kinect Beta

Ohhh. Thanks.

Re: the rutt changes I was mentioning... I just like having blend modes/depth testing independently of the line modes, built in rotation/translation matrix (which threw me for a loop at first because of the y/z stuff in the rutt code).

In thinking about it more... is there a reason you keep blend and depth automatic? Is it in the style of the O.G rutt etra to do that?

coebro's picture
Re: v002 Open Kinect Beta

Beautiful! :) looks very very good

vade's picture
Re: v002 Open Kinect Beta

3.0 in dev is muuuch faster and exposes many more options such as blend, depth, texture coordinate generation options, faster normals (and oneshot rendering) less memory usage if you don't need normals, etc. Maybe even some new tricks :)

As for culling I have some ideas :) I played with it last night in vdmx b8 for hours at the nightclub, seems to work well.

gtoledo3's picture
Re: v002 Open Kinect Beta

I have written clip planes into a version since I wrote this, but haven't compiled it yet... it seems like a reasonable approach to me, but I'm a little unclear about maximizing depth buffer resolution and how tweaking clip planes changes that. I think I need to look into that more, so I understand the ins and outs of that better. It's kind of interesting to be able to have multiple clip planes.

I've also experimented with adding in a port for normal texture, in addition to your normal calc stuff (which is handy to me, because I would prefer to prep the normal map sometimes myself). I lopped out all of the implicit depth/blend mode stuff that's called on w/ the different wireframe/point modes, and have that working as separate ports, with some extra blend modes from usual.

So, have you found that the changes in texture handling in the last one weren't actually a big improvement? I was wondering about that.

vade's picture
Re: v002 Open Kinect Beta

Hm?

gtoledo3's picture
Re: v002 Open Kinect Beta

I didn't phrase that clearly.. in the newer one you use the shader files and shader helper classes, in the first you didn't.

I guess I was just wondering what it is that generally that makes for such a big speed boost you're talking about with the latest unreleased one.

vade's picture
Re: v002 Open Kinect Beta

Leveraging newer GL extensions that allow me to use Multiple Render Targets ; ie, gl_FragData[0], [1], [2] for one shot rendering of all verts, normals and tex coord in one FBO, in one pass, with one shader. Vertex array objects (vaos) which encasulate opengl state vectors so enabling client state and binding buffers does not force a state validation pass. Unfortunately currently neither are exposed in QC in any meaningful way.

gtoledo3's picture
Re: v002 Open Kinect Beta

What would be the right way to invoke stippling w/ the Rutt Etra? It's kinda driving me nuts...

I have it working, but not without errors in console. I've tried all kinds of permutations of where to have it happening...doesn't seem to make much of a difference, other than when I've had stippling not working at all.

What happens is that I get an error with with the texture binding...

2/22/11 12:32:30 AM Quartz Composer[8807] *** OpenGL error 0x0500 (invalid enumerant) in function "-[QCPlugInInputImage bindTextureRepresentationToCGLContext:textureUnit:normalizeCoordinates:]"

or

2/22/11 2:07:23 AM Quartz Composer[9614] *** OpenGL error 0x0500 (invalid enumerant) in function "-[QCPlugInInputImage unbindTextureRepresentationFromCGLContext:textureUnit:]"

It actually does work and achieve the stippling look fine, it's just that it dumps these errors to console, and gets choppy when the errors happen. The build without any stippling stuff seems fine.

This is the zip, with all of the stippling stuff commented out, so that no stippling/errors happen. I've only tried stippling with the line modes, since I haven't got that working without error yet.

This isn't 100% reflective of all the ways I've tried to get this going, and has some redundant places where stuff is commented out (more than one place where stippling stuff is invoked needlessly). I've tried it before and during vbo drawing, and it doesn't seem to make a difference with errors. I've also had a separate boolean to entirely disable all stippling stuff, but that doesn't seem to make a dif. I've also tried separating all of the stippling enabling ala:

if(self.inputStipple == FALSE)
      glDisable(GL_LINE_STIPPLE);
   else
   {
      glEnable(GL_LINE_STIPPLE);
      glLineStipple(self.inputStippleCount, self.inputStipplePattern);
 
   }
 
   if(self.inputStipplePattern)            
   {   
      glEnable(GL_LINE_STIPPLE_PATTERN);
 
   }
   else
   {
      glDisable(GL_LINE_STIPPLE_PATTERN);
 
   }
 
   if(self.inputStippleCount)            
   {   
      glEnable(GL_LINE_STIPPLE_REPEAT);
 
   }
   else
   {
      glDisable(GL_LINE_STIPPLE_REPEAT);
 
   }

...which isn't in this attachment, but also doesn't make a difference.

The stippling looks really cool to me, so I'd like to get it working. Thanks for any thoughts. For my "peanut gallery" vote, I'd like to see stippling, and maybe translation, rotation, depth testing, ala the add-ons in this, w/ the improved version of the Rutt Etra you've mentioned. (For any lurkers who can't help themselves from compiling this, all credit for anything good in the code goes to Vade/Bill Etra, anything sucky is on me... please just move on, and use v002.)

PreviewAttachmentSize
heightfield w stipple commented out.zip3.12 MB

gtoledo3's picture
Re: v002 Open Kinect Beta

I'm breaking this off into a separate topic, since this should really be about the kinect. Sorry!

toneburst's picture
Re: v002 Open Kinect Beta

This may have already been mentioned, but any chance of having simultaneous outputs for the Color and IR cameras, rather than a switch to flip between them? I can't think of any obvious use for such a facility, off the top of my head, but it would add flexibility, and I'm sure there are scenarios where having access to both images at the same time would prove useful.

Anyway, keep up the great work!

a|x

waxtastic's picture
Re: v002 Open Kinect Beta

+1 for this.

I would also like to see these:

  • An option for inverting the depth image to go from white to black
  • An absolute tilt option in addition to the current one that seems to be linked to the accelerometers.

dust's picture
Re: v002 Open Kinect Beta

any luck getting multiple devices working ? great depth image.

vade's picture
Re: v002 Open Kinect Beta

Unfortunately, that is a device / bandwidth limitation from my understanding. Its one or the other.

vade's picture
Re: v002 Open Kinect Beta

The next beta has an option to flip the depth image. I'll see about the tilt, but I think thats how it works (at least with libfreenect).

vade's picture
Re: v002 Open Kinect Beta

I only have one kinect to test with, so its hard for me to know if its working. Ill see what I can do.

gtoledo3's picture
Re: v002 Open Kinect Beta

Kineme Kinect Tools can address multiple...

There's a powering the kinect from the usb bus issue, but if you handle that on your own (via doing a card to USB adapter or whatever, depending on what kind of computer you're running the answer would be different I guess), multiple can work. I think multiple can even work without totally correct power (one of the unit's led will stay yellow), but I suspect it may be the cause of burned out power supplies eventually.

...or do you mean that the quality of image you're outputting precludes multiple somehow?

gtoledo3's picture
Re: v002 Open Kinect Beta

Have you posted the project? I'd like to take a stab at it if you have...

borego's picture
Re: v002 Open Kinect Beta

Hey all,

Have there been any updates to this patch since posting? It looks like most of these replies/etc are from 2011. We're working on building a composition running 3 Kinects on 3 separate monitors, and using the v002 patch over the Kineme patch would be preferable, so allowing multiple Kinects would be great.

Thanks!

EDIT: Looks like I should've tested it before I posted - multi-Kinect works. Whoops!

bangnoise's picture
Re: v002 Open Kinect Beta

The repo for this is now public if you want to build the very latest yourself - https://github.com/v002/v002-Open-Kinect

kimba23's picture
Re: v002 Open Kinect Beta

looks like it was updated recently https://github.com/v002/

bangnoise was faster :)