Kinect Known Issues....

gtoledo3's picture

I was looking at the kinect known issues...

Is the ten-ish pixel strip of alpha at the right side of the Depth Map a known issue (the grey strip in the pic at the right edge)? Does it do that on other units?

That, the rgb fringe of a pixel or so at the edges, and the distance of the depth sensor from the rgb cam, makes aligning the rgb image with the depth capture somewhat imprecise.

I'm attaching some pics of captures for fun...

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

gtoledo3's picture
Re: Kinect Known Issues....

In preview here the strip will show up as black, not grey...

dust's picture
Re: Kinect Known Issues....

it seems you get the same fringe artifacts as well. i really first noticed it in ofx on the rgb image but its there in qc as well. it doesn't seem to bother the tracking.

tracking seems to be working in qc. here i'm overlaying the depth blob image with my isight which are at two different depths.

SteveElbows's picture
Re: Kinect Known Issues....

My Kinect should be coming today. I was just wondering if there is any idea when the new driver with better framerate might make its way into kinect tools? Cheers.

gtoledo3's picture
Re: Kinect Known Issues....

Sure- it's definitely close enough of a match for virtual object placement; it's using the depth map to create a mesh then texturing the result that gets a little awkward... but doable.

I'm sort of curious about the "strip" on the depth map on the right side.

gtoledo3's picture
Re: Kinect Known Issues....

I'm curious about that as well. Also, I'm sort of curious about what looks like the queueing of the frames. What I mean by that, is that if I have frames rendering, and do something that significantly changes the graph, or causes the Editor to bog down (like transmutating a macro), one sees all of the frames that the kinect was grabbing while the editor was hosed suddenly render quickly.

It looks kind of funny/cool... it's the only time I can remember ever seeing anything like that happen with QC.

SteveElbows's picture
Re: Kinect Known Issues....

Wahey my Kinect arrived :)

Saw that there was sourcecode for the plugin so I thought Id have a go at patching it myself but got horribly stuck, cant build in xcode, not sure if its because SkankySDK.framework is red.

dust's picture
Re: Kinect Known Issues....

I can say the strip on the right side doesn't seem to be there in ofx. I'm thinking it just has something to do with the depth camera being offset and that may be just the inter oculars disparity between rgb and d images. kineme makes the alpha channel clear where there is no depth reading. you got to work around that. it seems color blob tracking tracking works but when you track with open cv the alpha becomes a problem. yes I did notice using the mesh depth field mapping to be little off. although gt your depth field test images look very good. I have noticed if you correct the offset for the right side the left side is off and vise versa. so thinking there must be away to double up the depth image and offset for both sides ? either way nice test images. are those coming from you cl kernel

smokris's picture
Re: Kinect Known Issues....

The framerate improvements described in https://github.com/OpenKinect/libfreenect/issues/issue/22 aren't working for me --- if I make those changes, libfreenect never receives a valid frame from my Kinect.

Still investigating.

gtoledo3's picture
Re: Kinect Known Issues....

The fact that the strip is there isn't hard to workaround; I wanted to make sure it was coming through that way in QC for others, to make sure my unit wasn't out of alignment.

I find myself shifting to the right, and oversizing the RGB slightly to make it match. I haven't found any magic number and I don't think there really is one, given that the RGB channel is to the side of the depth sensor. It becomes a bit like trying to match a left and right eye image.

Yes, that is essentially the OpenCL kernel that I posted with R/G/B inverted but not alpha, and some other tweaks. I've had to add steps to offset and resize the color image and keep origin 0,0.

Though I've been visualizing the output mainly with OpenCL in my tests, I have to throw in that this is a very good scenario to use the Kineme3D heightfield for. The K3D heightfield's resolution isn't locked to pixel w/h, and can go higher than the GLSL grid in vertex count. There are already separate image inputs for the heightfield and the texture images. You can use the built in K3D deformers on the result, and stuff flies along. However, I like that I can get the vertex and color info with the OpenCL kernel and do_stuff.

gtoledo3's picture
Re: Kinect Known Issues....

Thanks.

Maybe I'm wrong, but it seems like part of the problem is that it isn't dropping frames. I may be using the wrong lingo.

Whenever something happens that normally bogs QC down a bit (taking the editor window and vigorously shaking it back and forth, moving a patch back and forth quickly on the editor, doing a screen cap, taking focus on or off of QC, etc.) I can see latency increase, and I'm suddenly still seeing the frames for X frames ago, instead of the signal just being "now" + normal latency.

However, if I just let stuff render, don't do a bunch of editor events, etc., I see very minimal latency, and performance seems to stay at a really quick rate in general. It's when what looks like queuing of the video stream starts happening, that stuff goes awry. If too much of that happens (err, queuing of the video stream, or storing in memory, or whatever is going on exactly...not sure of the right technical term), then QC will actually crash.

Again, thanks for whipping this up.

gtoledo3's picture
Re: Kinect Known Issues....

The new release is slick.

gtoledo3's picture
Re: Kinect Known Issues....

It's way quicker. It's so fast on my laptop that I'm able to make it look like my room and I can dance in realtime. This thing is actually pretty fun, especially now that the performance is getting sweet.

SteveElbows's picture
Re: Kinect Known Issues....

Thanks so much for the updated version, its great :)

SteveElbows's picture
Re: Kinect Known Issues....

George Id just like to thank you very much for the various OpenCL compositions you posted in the past that work nicely with the Kinect. I had fun playing around with this stuff on Saturday evening. Ive yet to create anything very dissimilar to the videos youve already posted, but if I do I will post video.

SteveElbows's picture
Re: Kinect Known Issues....

Perhaps the most simplegeek fun I had was when I realised I could remove the background by simply placing a large qc cube at the required z-pos.

vade's picture
Re: Kinect Known Issues....

Ha! Nice. Great track selection.

dust's picture
Re: Kinect Known Issues....

check out this vade and coge webkit kinect fun.

gtoledo3's picture
Re: Kinect Known Issues....

That song is pretty awesome. I feel like Timbaland's production on that was great. It was a real interesting bridge between what was sonically trendy at the time, classic stuff, and then things that have come to be associated with his sound since.

Ginuwine does some move in the video, and then they're riding an electric bull... the pace my room was moving somehow reminded me of that movement from the vid.

gtoledo3's picture
Re: Kinect Known Issues....

You can use a sprite behind the main mesh renderer too... you can link the color of the sprite to your clear color and make width and height really large.

gtoledo3's picture
Re: Kinect Known Issues....

That's cool. Please do!

If you change the mesh renderer to point mesh, and make the points kind of chunky, it will look like a smooth mesh with gaps on the sides - if it's not using GL Lighting.

I've had a few functions I'm trying to get into the kernel. I've come close adding an alpha threshold function to the color output. Last night I got it working to change all of the color outputs by a given r/g/b/a input value, so now I just need to add the "if vector length is less than rgba, then 0,0,0,0" aspect to it successfully.

I was trying to add a certain smoothstep function, but when I write "smoothstep" it doesn't highlight in the OpenCL editor as a function. Maybe I'm using it wrong(?), but it looks like it's part of the CL 1.0 spec.

I want to actually figure out how to pass only vertices that exceed a certain floor value of the color vector length, but my syntax seems to not work, and the CL editor doesn't highlight any lines, it just tells me there was a compiler error. It becomes a little trying! I've never output "less that what I put in" with any kernel.

I guess on the plus side, I've learned a ton more about OpenCL.

SteveElbows's picture
Re: Kinect Known Issues....

Point meshes are what I have been playing with so far, as Id done a bit of that stuff in the past with other CL things. But Ive not had much time and havent achieved anything I like yet, will be trying again tonight or tomorrow. Cheers.

gtoledo3's picture
Re: Kinect Known Issues....

I feel like every time I try to work on a kernel, I'm relearning things. I finally got my syntax correct on at least a couple things I was attempting, so with some polish, it will be quite cool.

Man, I wish I could just type pseudocode sentences of what I want to happen... it's always a pain when I just forgot a semi-colon or parenthesis somewhere. The compiler doesn't always catch that. Also, it seems like things that should work sometimes just don't, and it gives a message like "compiler error", with no useful lines highlighted.

I was trying to do a smoothstep function last night, and it didn't highlight like it's able to be done, but I'm not sure if I'm using it correctly - it should highlight either way, it seems like. Anyway, things like that sometimes make it a pain to get kernels going. "It is me compiler, or is it you?"

dust's picture
Re: Kinect Known Issues....

the smoothstep is a normalized hermite interpolation.

it works like this. your input value gets stepped in a normalized 0->1 if start vale is less than the input value and input value is less than end value otherwise the result is 0 or 1

so this is sort of pseudo code i guess or well the kernel set up needs to be done but this is basically the smooth step function.

float startedge = 0;
float endedge = 1;
float inputvalue = .5;
float stepvalue = 0;
 
float step = clamp(((inputvalue - startedge) / (endedge - startedge), 0, 1);
 
stepvalue[tid] = step * step * (3 - 2 * step ); 

gtoledo3's picture
Re: Kinect Known Issues....

Sure, but I mean that smoothstep is a supported function in opencl 1.0 but it seems not to be in QC. I wasn't meaning that someone couldn't write something equivalent. I was using smoothstep as an example.... I could have picked any of a number of other functions or operations ( I'm not sure if I'm using the right semantics) that seem like they should be supported but may not be.

dust's picture
Re: Kinect Known Issues....

thats actually the open cl smooth step function i converted from pseudo the smooth step will not return anything but one or zero unless the input value is between 0 and 1. so using it to blur or gradient is possible but you cant lets say input your mouse coordinates and normalize them from 0 to 1 because qc unit is -1 to 1 but you can convert units to pixels then subtract width or height depending on which axis x or y and then you can use them with smooth step.