KinectTools 0.3 source, or, getting a float image out of KinectTools

vade's picture

Subject says it all.

I want a higher fidelity image out of the depth map, right now its ARGB 8 bit/channel, and I know float data is in there.

Is the source available, or should I go about making my own Kinect driver plugin with thresholding, depth map normalization etc?

Just curious, I have some uses for this and need some more control, the source for the latest would be helpful, and save me some time in developing my own public API solution.

gtoledo3's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

I'd really like to see some integration of Avin's driver/OpenNI&Nite basic stuff. You can't really use libfreenect driver and the openni(avin's branch) driver at the same time unless there's something quirky going on with my install (and I don't think that's the case).

How did you do the bit test? Did you get point positions in OpenCL, sample, then do maths to see if step values were different enough? Something easier? (sorry for the barrage).

It would be useful to have the user tracker, push gesture, and hand cursor implemented as openCV haar tracker type stuff for QC. (Sorry, not trying to get OT from your main thrust of bit depth, it's that expansion of kinect stuff has been in the back of my mind.)

toneburst's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

Floating-point depth image sounds good to me, vade.

a|x

cwright's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

and I think it might even be possible (unfortunately, I'm not in a position to experiment with it anymore... alas :/)

vade's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

So I have this working, but its slower than the Kineme one. Looking into the issue of why, it looks like a huge issue is not having vanilla RGB QCPluginPixelFormat options (no alpha). Being forced to convert to ARGB, and uploading ARGB (for the color image) is costing me a lot of CPU.

I am using a background thread, and the asynchronous freenect callbacks to write to the buffers I am using, and I am using vImage / Accelerate.framework for the RGB -> ARGB (vImage has no BGRA conversions, which would speed things up immensely for Intel).

Looking at the Kinect Tools 0.2 source via Skank SDK, it looks like Skank SDK plugins can asynchronously update ports values individually, whenever the callback happens (setImageValue:). Public API plugins cant do that, you can only do it when executeAtTime is called, and then you have to update all of your image ports at once.

The public API for QC really lacks a lot of stuff that would give people control, and being able to asynchronously set port values outside of execute at time, whenever events in the real world happen would be amazingly useful.

A few additional Pixel Formats would eliminate conversions (even though they can be made fast)

Grumble

This is the kind of stuff that makes me so annoyed.

** Assuming I have not done anything tremendously stupid to cause these slowdowns. Bleh.

vade's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

Yup, commenting both of my outputImageProviderFromBufferWithPixelFormat factory methods to not actually output the images (but continuing all processing, capturing asynchronously on the background thread) almost halves my CPU usage from ~ 65% CPU to maybe 35-40. Definitely looks like some improvements could be made there? Im going to try going direct to GL to see if this avoids the issues, but, goddamn. So much time spent in mach_msg_trap when I use those helper methods.

cwright's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

Quote:
"Give yourself to the dark side. It is the only way you can save your [performances]"

It's a shame you signed up for the law-abiding team ;) And your handle is just 1 letter away too!

vade's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

Ha!

Amusingly, handling the texturing myself, and using the outputImageWithTexture (vs Buffer) saves me 15% CPU over outputting a buffer provider and letting QC do the texturing for me.

I can live with that. Grumble grumble.

Edit: amusingly, if you use the GL route, you can trick QC into letting you upload an RGB image masquerading as a RGBA one, and save yourself a CPU side copy.

Suck it QC.

gtoledo3's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

Still using libfreenect driver?

dust's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

haha good one chris. "vader"

vade's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

Yeah. Admittedly I'm behind on whats hot in kinect land. Is the OpenNI driver faster/better?

I compiled the latest libusb from scratch, then libfreenect latest. Seems decently fast, my issues are kind of in QCs public API land. Currently i have it running as fast as the Kineme Kinect tools after putzing with output image trickery.

vade's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

So you are "thejokeexplainer" on reddit? ;)

dust's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

yeah thats me on reddit. somethings just go over peoples head and need to be explicitly spelled out ;)

dust's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

this is very interesting to me vade. i like the idea of the libfreenect. open ni is defiantly interesting. the skeleton tracking is hot right now. for me i need a basic implementation of just rgba and a depth image.

i managed to get a plugin working from your ofx 2 qc plug project. i don't think grabbing the images off screen in ofx and sending to qc is the best solution so i would be interested in an official plug for qc.

would most defiantly donate some loot if you made this available for use.

have made some interesting things with some of your plugins vade, should i send you a message to you via your site. i'm thinking a few of them i would like to release commercially and i noticed you use creative commons lic.

vade's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

If you got ofxQCPlugin working then you would know it was not rendering anything 'offscreen' and "sending" it to QC, it works natively within QC as a real legit plugin, and is able to pass ofImages and ofTextures in and out of image ports in plugins as a regular 3rd party plugin does.

There is nothing 'unofficial' about it. It works exactly like any run of the mill official API plugin. Your comments about this always confuse me, to be honest.

At the moment not super interested in the OpenNI skeleton tracking and interaction stuff, so I don't expect anything like that to be added in there, but basic capture is already up and running, I just need to add color image offset, thresholding and calibration for the depth image now.

As for licensing, lets take that off of Kineme.net

dust's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

hmm i was talking about how i was using an ofx screen grab to get the ofx image as not being the best method not anything your code is doing. yes ofx is legit and i know your plugin uses official stuff don't want there to be any confusion. just wanted to let you know, i wanted to use some of your code if possible.

its interesting if you move the qc editor you can actually see the the ofx app running. sort of glitchy haven't seen this with any thing else but the kinect. the kineme plugin sometimes does a similar glitch as well when you move the editor..

sent an email....

gtoledo3's picture
Re: KinectTools 0.3 source, or, getting a float image out of ...

When the libfreenect driver is addressing the kinect, it seems to preclude OpenNI from accessing it as well.