v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

One thing to note, the Rutt Etra 3.0 does not (currently) replace the older plugin. So your old comps will not inherit the new plugin - this is for compatibility and testing. Once I'm comfortable things are stable and sane, I will switch it over.

The more you know™

Both awesome updates, thanks vade!!!

great, thanks !

Thanks!

Did you wind up feeling like pitch/yaw/roll/w(err, I'm forgetting what the tech term for this is) was odd for clip planes? I see you only implemented z.

If you get a chance, it would be cool to see how the kinect project works; I'm curious about outputting intensity image. Thanks either way, it's awesome work!

The speed different on the rutt etra is incredible! and Kinect rectification :)! muchas gracias. brilliant work vade

@gtoledo3 well, I only implemented it in Z because it was seemed to make the most sense. What do you mean about outputting intensity image?

For the depth? The reason I do not render the depth map as Intensity any longer is that Intensity images are not "renderable" with FBOs in a way that works consistently on all GPUs that I could find. In order to do the rectification and re-projection on the GPU I have to use a renderable texture attachment format There are other extensions, like GL_RED to render single channel colors, but, as you might expect, rendering a GL_RED texture to a full color buffer without using a shader, renders, well, Red, rather than intensity.

Internally, the texture uploaded to the GPU is the full range ushort format depth image straight from the Kinect, as normalized float (I multiply x 65535 in the fragment shader to get the full range and logic back, although I suppose I could do some sort of integer texture, but there is no need). I output a 32 bit RGBA float image for compatibility (QC seems to like RGBA more than Intensity for post processing), but uploading as Intensity keeps things fast, internally, if that is your concern.

@itsthejayj Yes, the speed difference is quite large. Using MRT for FBOs is a huge win, as is having optimal buffer formats so the GPU will not do expensive expansion converting types. Subtle, subtle things make huge performance differences!

As per Kinect rectification, Kyle McDonald is working with the Open Kinect/libfreenect people to port some OpenNI code that reads the internal calibration data from the Kinect, so rectification will be dead on in the future (nice!)

Re: the kinect patch, I wasn't actually aware you had changed what you were doing, I hadn't tested it yet. I really liked the results of what you were getting with the last one, save for maybe the dark=close aspect - the resolution of depth steps was great. If you've achieved that in a more stable way now, even better.

When I put 6 clip planes on a heightfield, it's pretty ridiculous, but I do think that given kinecty activities, that 4 become really advantageous, or at least capability to do more than near and far z clip. The reason for that is that one can carve out the actual "floor" in the depth map representation of a room setting if one can position a clip plane in any rotation. This allows one to get nice outlines of people, or even get better tracking of people at a given distance, because the floor can be carved out. You can "sorta" do it with cropping the depth image ahead of time, but it's more accurate result-wise to clip the mesh after it's been extruded.

What I really wanted to do when I was messing around with your project (thanks for publishing that by the way), and adding in clip planes was to have a setting where more clip planes could get added or subtracted by a little + or - tab. I've not done that before though. I also really wanted to change the stock gl rotation scheme to qc type rotation, so it would make more sense to people to use. Just some thoughts.

Another thought I've had in this line is adding a port to be able to plugin arbitrary vertices other than grid. Distorting a sphere is really cool, but so is a cube... so then I go "where does it stop"? I can see some cool effects coming out of that, though I don't know how great performance would be with large input structures.

The depth map has the exact same detail level, I am not lowering the bit depth of the image, so there is no loss there.

You can do arbitrary shapes now with position map, pass in the positions you want for a vertex (pixel) in RGB and it will draw it in XYZ. Same input port.

Re the extra clip planes, I totally understand their usefulness. I might consider adding them.

What do you mean stock GL rotation vs QC rotation?

That's really great about the rgb image to 3d object. I haven't really investigated. I guess I was thinking about the effect one gets when an extrusion glsl shader is run as an environment and it's wrapping an arbitrary object. I've wondered about using a structure of vertices (rather than image) to define the base geometry. What you're doing may make way more sense... It seems like two different things/likely resulting visual effects, maybe.

What I meant was how when you specify a clip plane and then the "gl equation" for it as (0,0,0,0) for example, and then manipulate those numbers raw, how they clip plane moves around in a pitch,yaw,roll type way for the first 3 coords, and then the 4th lane pushes it front to back, to my memory at least. It's been a while since I've messed with this.

That defines the plane equation, so its not quite "pushing it back", or even pitch, yaw, role for those input params, its specifying parameters for the plane equation you pass to OpenGL, which does not really work as intuitively as pitch, yaw, role:

Something like, where the 3 vec3s are points defining a plane (which is how I am using it).

static void planeEquation(float x1, float y1, float z1, float x2, float y2, float z2, float x3, float y3, float z3, double* eq)
{
    eq[0] = (y1*(z2 - z3)) + (y2*(z3 - z1)) + (y3*(z1 - z2));
    eq[1] = (z1*(x2 - x3)) + (z2*(x3 - x1)) + (z3*(x1 - x2));
    eq[2] = (x1*(y2 - y3)) + (x2*(y3 - y1)) + (x3*(y1 - y2));
    eq[3] = -((x1*((y2*z3) - (y3*z2))) + (x2*((y3*z1) - (y1*z3))) + (x3*((y1*z2) - (y2*z1))));
}

Where eq[0, 1, 2, 3] is what is passed to define the clip plane. The normal of the clip plane determines if you clip "in" or "out"

Oh, ok. Thanks vade...

Wow, so you aren't using one vec4 to define the clip plane, but 4 equations. I have to wrap my mind around that/try it actually, I haven't seen any examples that have used that approach before. Cool.

My understanding is you have to do this if you want arbitrary planes to be able to be defined, as a single vec4 is not enough to define a plane. I could be wrong about how it works, but I've seen the above approach used elsewhere, and it seems correct.

Edit: and to be pedantic, I am using the output of the plane equation, im not sending in 4 equations, that code is simply populating the array eq with the correct floats, which is then sent to GL.

thanks vade. btw, is the motor supposed to turn on and look up vertically? can't seem to control it.

Ok, I'm not making any assertions here. I only know what I've read, and I certainly put as much or more stock in your take.

If you are using four equations to create your floats, that then populate the gl clip plane equation, which is a vec4, then by definition, a vec4 is enough to define a plane. I think the way your equations is allowing you to specify the planes is probably more intuitive than just plopping in numbers.

There are many examples that show the clip plane "vec4" equation having floats directly input into it, which will clip, far, near, or any of the four quadrants of whatever the area in question is, without doing any kind of conversion, just plopping the float values into the equation. Because of that, and the fact that inputting numbers into the equation seems to work in a logical/predictable way to me, never made me think much more about actually formatting the numbers more.

It's really trivial to have the "plane" be parallel to the mesh, then tweak X (err, I'm calling the first lane of the equation X in this case) and have it cut off at a bit of an angle (to my memory) as if a plane is being rotated in X. I seem to remember that tweaking W pushes the clip plane back and forth, allowing one to clip far and near. I guess it's not pitch, yaw, roll really, but it kind of feels intuitive to me in the same way that manipulating gl look at does.

I'm excited to try out that method you're using and see how it works the plane. (If you end up recalling where you saw the clip plane approach you're using outlined, it would be really cool to be able to read up on it more, because it looks more advanced than examples I've seen.)

My claim was that a single point does not define a plane. a vec 4 interpreted as a plane equation does. Clearly, the API takes a vec4, my assertion is not that it does not, is that the vec4 is not treated or conceptually the same as a "vertex" of a "mesh".

If you treat the inputs like a point/vertex, it wont work as you want it to. If you want to think about the clip planes as "points" mentally, you have to use a plane equation otherwise, how else will you do it. That was my only "point", pun intended.

The Tilt input controls this. Its worked for me, and others. Is it not working for you?

For comparisons sake, take a particular line in 2D space. We can define it mathematically with two (unique) (X, Y) co-ordinates: {(x1, y1), (x2, y2)} or we can define it with a general line equation y = a.x +b (where '.' means multiply ;-]) and the particular values for {a, b}.

eg. Line passing through points: {(-2, 0), (0, 4)} is equivalent to y = a.x +b where (a, b) = {2, 4}.

The second method is less data to store/pass by factor of 2 but requires an understanding of what {2, 4} means ie. it's not just an (X, Y) point defining a line in 2D space.

I haven't worked through Vades equations to see what kind of function is going on there (in terms of how the clip plane vec4 is actually interpreted by OpenGL) , just clearing up what may be a somewhat semantic issue — apologies if you already get this.

The rectangular grid option does not appear to be responsive to the element size parameter, unlike the scanlines, points, and triangular grid options.

Generally a very nice refinement of the plugin!

hi, we're encountering a strange problem with a custom QC using v002 Open Kinect : on one computer everything works fine; on another computer, with same Mac OS X 10.6 version, same composition, same plugins and patches in Library/Graphics and same kinect device, when we use Depth Map as Depth Format + Infrared as Image Format, we can't see anything. If we use Color as Image Format however, the camera image displays properly.
Any idea ?