v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

vade's picture

Hello.

Im happy to share with you 2 new beta projects I've been working on..

First up,

v002 Open Kinect 1.0, Beta 2

10.6 only.

Color and IR images.

Depth Image in Floating point.

Optional Rectified color image - meaning re-projection from current depth buffer so color image is mapped appropriately onto the projected depth location. This uses computed averages from calibrated kinects. This means it may, or may not, be spot on, since this skips the calibration phase and uses some 'best guess' numbers. Only in Color mode (IR image is always "rectified" by design).

New Position map output mode. Output a 32 bit RGBA image of XYZ location data for rendering with OpenCL, or with the new v002 Rutt Etra 3.0 in XYZ mode.

Optional Depth unprojection, correcting for FOV issues with Kinect camera, allowing for proper depth map in 3D space without sizing or scaling issues (only in Position map mode).

Color image and depth image rectification is performed on the GPU, for ultra fast results, using FBO + MRT single shot rendering.

v002 Rutt Etra 3.0.

10.6 only

Whats new:

Much faster, totally re-written render backend, leveraging multiple render targets for single shot rendering of multiple assets (FBO + MRT again). With a decent machine, you can render pixel for vertex 1080p content at 30 fps.

Raw RGB to XYZ mode: Rather than supply a luminosity "displacement" map, you can instead supply an RGB position map, allowing for rendering of any shape.

Color Correct - Applies only to depth map input. Useful for specifying depth map as "data" rather than as image. Off by default. This may go away, or be assumed to be on when in luma mode, and off when in XYZ mode.

Color input for tinting rendering, good for controlling alpha when rendering with point sprites.

Point Sprite support. Supply a GL_TEXTURE_2D image into the point sprite image input port. Use the Image Texturing properties to force GL_TEXTURE_2D and mip map. Suggest using a small image, less than 64x64.

Clipping: Set near and far clipping planes to remove background and foreground objects when rendering Kinect position maps, or to focus on specific luma levels of interest.

Texture modes: Some additional texture mapping modes. Spherical mode when generating normals creates a nice, liquid feel, for example.

Point attenuation. (experimental at this stage).

Independent depth, blending, position, translation and scaling controls.

Downloads:

http://v002.info/downloads/beta/v002%20Open%20Kinect.plugin.zip http://v002.info/downloads/beta/v002%20Rutt%20Etra%203.0.plugin.zip

Example "Kinectra" patch: http://v002.info/downloads/beta/Kinectra%20.qtz

Licenses

v002 Open Kinect uses libfreenect, thus is Apache 20 licensed.

v002 Rutt Etra is the standard CC-NC-AT-SA license.

Known issues: Some oddities with v002 Open Kinect in VDMX.

Mad props to Kyle McDonald for help with Kinect rectification.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

vade's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

One thing to note, the Rutt Etra 3.0 does not (currently) replace the older plugin. So your old comps will not inherit the new plugin - this is for compatibility and testing. Once I'm comfortable things are stable and sane, I will switch it over.

The more you know™

monobrau's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

Both awesome updates, thanks vade!!!

franz's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

great, thanks !

gtoledo3's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

Thanks!

Did you wind up feeling like pitch/yaw/roll/w(err, I'm forgetting what the tech term for this is) was odd for clip planes? I see you only implemented z.

If you get a chance, it would be cool to see how the kinect project works; I'm curious about outputting intensity image. Thanks either way, it's awesome work!

itsthejayj's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

The speed different on the rutt etra is incredible! and Kinect rectification :)! muchas gracias. brilliant work vade

vade's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

@gtoledo3 well, I only implemented it in Z because it was seemed to make the most sense. What do you mean about outputting intensity image?

For the depth? The reason I do not render the depth map as Intensity any longer is that Intensity images are not "renderable" with FBOs in a way that works consistently on all GPUs that I could find. In order to do the rectification and re-projection on the GPU I have to use a renderable texture attachment format There are other extensions, like GL_RED to render single channel colors, but, as you might expect, rendering a GL_RED texture to a full color buffer without using a shader, renders, well, Red, rather than intensity.

Internally, the texture uploaded to the GPU is the full range ushort format depth image straight from the Kinect, as normalized float (I multiply x 65535 in the fragment shader to get the full range and logic back, although I suppose I could do some sort of integer texture, but there is no need). I output a 32 bit RGBA float image for compatibility (QC seems to like RGBA more than Intensity for post processing), but uploading as Intensity keeps things fast, internally, if that is your concern.

vade's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

@itsthejayj Yes, the speed difference is quite large. Using MRT for FBOs is a huge win, as is having optimal buffer formats so the GPU will not do expensive expansion converting types. Subtle, subtle things make huge performance differences!

As per Kinect rectification, Kyle McDonald is working with the Open Kinect/libfreenect people to port some OpenNI code that reads the internal calibration data from the Kinect, so rectification will be dead on in the future (nice!)

gtoledo3's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

Re: the kinect patch, I wasn't actually aware you had changed what you were doing, I hadn't tested it yet. I really liked the results of what you were getting with the last one, save for maybe the dark=close aspect - the resolution of depth steps was great. If you've achieved that in a more stable way now, even better.

When I put 6 clip planes on a heightfield, it's pretty ridiculous, but I do think that given kinecty activities, that 4 become really advantageous, or at least capability to do more than near and far z clip. The reason for that is that one can carve out the actual "floor" in the depth map representation of a room setting if one can position a clip plane in any rotation. This allows one to get nice outlines of people, or even get better tracking of people at a given distance, because the floor can be carved out. You can "sorta" do it with cropping the depth image ahead of time, but it's more accurate result-wise to clip the mesh after it's been extruded.

What I really wanted to do when I was messing around with your project (thanks for publishing that by the way), and adding in clip planes was to have a setting where more clip planes could get added or subtracted by a little + or - tab. I've not done that before though. I also really wanted to change the stock gl rotation scheme to qc type rotation, so it would make more sense to people to use. Just some thoughts.

Another thought I've had in this line is adding a port to be able to plugin arbitrary vertices other than grid. Distorting a sphere is really cool, but so is a cube... so then I go "where does it stop"? I can see some cool effects coming out of that, though I don't know how great performance would be with large input structures.

vade's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

The depth map has the exact same detail level, I am not lowering the bit depth of the image, so there is no loss there.

You can do arbitrary shapes now with position map, pass in the positions you want for a vertex (pixel) in RGB and it will draw it in XYZ. Same input port.

Re the extra clip planes, I totally understand their usefulness. I might consider adding them.

What do you mean stock GL rotation vs QC rotation?

gtoledo3's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

That's really great about the rgb image to 3d object. I haven't really investigated. I guess I was thinking about the effect one gets when an extrusion glsl shader is run as an environment and it's wrapping an arbitrary object. I've wondered about using a structure of vertices (rather than image) to define the base geometry. What you're doing may make way more sense... It seems like two different things/likely resulting visual effects, maybe.

gtoledo3's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

What I meant was how when you specify a clip plane and then the "gl equation" for it as (0,0,0,0) for example, and then manipulate those numbers raw, how they clip plane moves around in a pitch,yaw,roll type way for the first 3 coords, and then the 4th lane pushes it front to back, to my memory at least. It's been a while since I've messed with this.

vade's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

That defines the plane equation, so its not quite "pushing it back", or even pitch, yaw, role for those input params, its specifying parameters for the plane equation you pass to OpenGL, which does not really work as intuitively as pitch, yaw, role:

Something like, where the 3 vec3s are points defining a plane (which is how I am using it).

static void planeEquation(float x1, float y1, float z1, float x2, float y2, float z2, float x3, float y3, float z3, double* eq)
{
    eq[0] = (y1*(z2 - z3)) + (y2*(z3 - z1)) + (y3*(z1 - z2));
    eq[1] = (z1*(x2 - x3)) + (z2*(x3 - x1)) + (z3*(x1 - x2));
    eq[2] = (x1*(y2 - y3)) + (x2*(y3 - y1)) + (x3*(y1 - y2));
    eq[3] = -((x1*((y2*z3) - (y3*z2))) + (x2*((y3*z1) - (y1*z3))) + (x3*((y1*z2) - (y2*z1))));
}

Where eq[0, 1, 2, 3] is what is passed to define the clip plane. The normal of the clip plane determines if you clip "in" or "out"

gtoledo3's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

Oh, ok. Thanks vade...

Wow, so you aren't using one vec4 to define the clip plane, but 4 equations. I have to wrap my mind around that/try it actually, I haven't seen any examples that have used that approach before. Cool.

vade's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

My understanding is you have to do this if you want arbitrary planes to be able to be defined, as a single vec4 is not enough to define a plane. I could be wrong about how it works, but I've seen the above approach used elsewhere, and it seems correct.

Edit: and to be pedantic, I am using the output of the plane equation, im not sending in 4 equations, that code is simply populating the array eq with the correct floats, which is then sent to GL.

gabemott's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

thanks vade. btw, is the motor supposed to turn on and look up vertically? can't seem to control it.

gtoledo3's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

Ok, I'm not making any assertions here. I only know what I've read, and I certainly put as much or more stock in your take.

If you are using four equations to create your floats, that then populate the gl clip plane equation, which is a vec4, then by definition, a vec4 is enough to define a plane. I think the way your equations is allowing you to specify the planes is probably more intuitive than just plopping in numbers.

There are many examples that show the clip plane "vec4" equation having floats directly input into it, which will clip, far, near, or any of the four quadrants of whatever the area in question is, without doing any kind of conversion, just plopping the float values into the equation. Because of that, and the fact that inputting numbers into the equation seems to work in a logical/predictable way to me, never made me think much more about actually formatting the numbers more.

It's really trivial to have the "plane" be parallel to the mesh, then tweak X (err, I'm calling the first lane of the equation X in this case) and have it cut off at a bit of an angle (to my memory) as if a plane is being rotated in X. I seem to remember that tweaking W pushes the clip plane back and forth, allowing one to clip far and near. I guess it's not pitch, yaw, roll really, but it kind of feels intuitive to me in the same way that manipulating gl look at does.

I'm excited to try out that method you're using and see how it works the plane. (If you end up recalling where you saw the clip plane approach you're using outlined, it would be really cool to be able to read up on it more, because it looks more advanced than examples I've seen.)

vade's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

My claim was that a single point does not define a plane. a vec 4 interpreted as a plane equation does. Clearly, the API takes a vec4, my assertion is not that it does not, is that the vec4 is not treated or conceptually the same as a "vertex" of a "mesh".

If you treat the inputs like a point/vertex, it wont work as you want it to. If you want to think about the clip planes as "points" mentally, you have to use a plane equation otherwise, how else will you do it. That was my only "point", pun intended.

vade's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

The Tilt input controls this. Its worked for me, and others. Is it not working for you?

usefuldesign.au's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

For comparisons sake, take a particular line in 2D space. We can define it mathematically with two (unique) (X, Y) co-ordinates: {(x1, y1), (x2, y2)} or we can define it with a general line equation y = a.x +b (where '.' means multiply ;-]) and the particular values for {a, b}.

eg. Line passing through points: {(-2, 0), (0, 4)} is equivalent to y = a.x +b where (a, b) = {2, 4}.

The second method is less data to store/pass by factor of 2 but requires an understanding of what {2, 4} means ie. it's not just an (X, Y) point defining a line in 2D space.

I haven't worked through Vades equations to see what kind of function is going on there (in terms of how the clip plane vec4 is actually interpreted by OpenGL) , just clearing up what may be a somewhat semantic issue — apologies if you already get this.

harrisonpault's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

The rectangular grid option does not appear to be responsive to the element size parameter, unlike the scanlines, points, and triangular grid options.

Generally a very nice refinement of the plugin!

emtv's picture
Re: v002 Open Kinect 1.0 Beta 2 & v002 Rutt Etra 3.0 Beta 1

hi, we're encountering a strange problem with a custom QC using v002 Open Kinect : on one computer everything works fine; on another computer, with same Mac OS X 10.6 version, same composition, same plugins and patches in Library/Graphics and same kinect device, when we use Depth Map as Depth Format + Infrared as Image Format, we can't see anything. If we use Color as Image Format however, the camera image displays properly.
Any idea ?