OSCeleton with QC (Kinect skeleton tracking via OSC)

waxtastic's picture

The very awesome OSCeleton basically uses the OpenNI NITE skeleton tracking and sends the joint positions via OSC to other apps. I got it to send QC compatible OSC messages via a liitle app I made in Max/MSP.

More info and download: http://mansteri.com/2011/01/osceleton-with-quartz-composer/

idlefon's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Cool !

Great toy for Animata lovers such as yourself.

monobrau's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Great!

I held a workshop with a similar kind of setup last week, instead of max/msp i've used processing to pass trough the osc info.

I'll post some more stuff once I've cleaned it up!

For the detailed install manual and source look here:

http://www.oneseconds.com/dev.html

cybero's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

it just had to happen :-)

mattl's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

this is amazing.. (both examples..) I would love if someone could explain exactly what the deficiencies of OSC receiver in Quartz composer are, that it requires a max/processing patch to convert the osc data..? Ive searched the forum archives and see several references to 2d structures etc but still arent really clear on what the problem is.. I'm trying to decode openTSPS osc data, and curious if setting up a similar patch would work.. any help would be great..

gtoledo3's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

The problem is that data has to be represented as indices, with either float, index, boolean, image, or vec4.

What the OSC of OSCeleton is sending is an indexed struct where each index contains floats or strings with spaces in between. The spaces are killing it for some reason (I believe), when it comes to being intercepted by QC.

I haven't had the time to look at all today, but it seems like what needs to happen is that the string needs to just get sent with something like & everywhere that there's a space, and receive as a structure of strings. The qcOSC patch would then be able to automatically configure to receive the data. That's the route I was thinking of and had been messing with yesterday when I went to bed (and awoke to saw you wrote the parser), but I was only getting one joint at a time, b/c it goes "2:arm leg hand" (sorta) w/ spaces between each name. So, you only get the first one.

The stock qc osc patch itself doesn't allow structure output/input.

I guess that the data could possibly be sent as one big string, if there were dividers that "made sense" for using string components to automatically convert the string to structure.

SteveElbows's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

I dont think its spaces that are killing it, its because different types of data are sent from OSCeleton in a single OSC message.

Been a few weeks since I looked at it but from memory OSCeleton is sending the joint name as a string, an integer that represents the user number, and then three floats for x y and z positions.

If OSCeleton had a mode where it put some of this as part of the message name rather than the values, then things would be fine in QC. I did actually hack the windows version of OSCeleton to do this, but my code is ugly. I asked the OSCeleton author about this stuff and he sounded receptive to the changes.

So as it stands OSCeleton sends messages such as /joint head 1 x y z (where head is a string, 1 is an integer, x y and z are floats)

For QC use we would like it to send something like /joint/1/head/ x y z (or /joint/head/1 x y z)

This will work ok because QC can receive multiple floats from a message and the resulting structure can be got at in the normal QC way.

monobrau's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

It seemed to me that OSCeleton sends info on one channel /joint, and then refreshes every independent joint by changing the string/x/y/z/user data, but it could also be that QCosc interprets it like this.

Before making the processing workaround I tried to make it work directly by making a structure and refresh independent joint segments by comparing the joint string and using a multiplexer to switch. It was too slow. After that I tried to simplify a bit with JS but stopped then. I've attached the comp's for reference. Maybe a full JS script could handle it faster? But a changed OSCeleton would be the best.

PreviewAttachmentSize
sensbloom.qtz34.82 KB
sensbloom3.qtz17.61 KB

SteveElbows's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

I started a discussion on the OSCeleton Google group about this stuff:

http://groups.google.com/group/osceleton/browse_thread/thread/666ee7f423...

Subtiv's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Hi,

does anybody know how we can use the kinect data both in Osceleton and in QC through the kineme plug in?

Right now when i try to do this i get "Segmentation Fault"...

gtoledo3's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

We can't have two drivers latched to the kinect, unfortunately. The same app that provides the joint data needs to provide the images.

Eventually, we really need something more like the actual user tracker (and maybe the hand tracker type stuff?) example wrapped in a patch of kinect tools.

SteveElbows's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

I believe there is something in OpenNI which enables a client-server model so that more than one app could access the stuff at once. However I doubt that all apps presently support this and I dont think it works on OS X yet either.

For QC I would suspect that the solution in future may be somebody writing a QC plugin that gives you depth map etc and the skeleton data from NITE at the same time.

SteveElbows's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

I did consider using 3rd party QC OSC plugin but I thought it would be a bit of a nightmare to handle the joint string in QC, which seems to be what you have discovered.

gtoledo3's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

I figured I would either figure out a way to parse the data using another app (like mansteri ended up while I was asleep! What a pleasant surprise to wake up to yesterday..) or to change the cpp of OSCeleton to put out data that is basically the 5 indices that qcOSC sees, but to try stuffing in & symbols (or something) everywhere there was a space. I don't know how reasonable that is, not having looked at the osceleton cpp much.

I know you posted something on the group about having figured out a workaround, where everything sends as a single input of a bunch of floats with spaces (I think this is what you were getting at) ... which surprised me, b/c I would have guess it needed to be string data in order to pass successfully with the spaces.

I think that it's probably better over the long run to lop out OSCeleton all together (no offense to the app), because it convolutes the process, and is unneeded, technically, to get the skeletal struct data into QC. You mention below about the server stuff - there are example apps that pump out the point structure (like point server) to terminal. It seems like something along these lines needs to just work in QC, and output the struct, ala how CV Haar Detector stuff works now, but perhaps without explicitly connecting a video object of any source, instead being an external provider (as opposed to processor).

monobrau's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Or maybe a Syphon implementation in OSCeleton. Oh I wish my programming skills were better...

gtoledo3's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

BTW, I'm still using a modded linux build of OSCeleton, not the "OS X version", which, for some reason, seems to take forever to latch onto the user.

This is the file I've been using to run it. (Thanks to Zachary Polley - he got this makefile/bin rigged up, and also to smokris/cwright/bmellen for catching and answering serious earfuls of Terminal questions from me this week.) It's been giving me better results than the official setup.

PreviewAttachmentSize
OSCeleton-bak 2.zip266.33 KB

gtoledo3's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

That doesn't actually make sense (please don't take that as critical), since Syphon is for pumping around images... it's essentially an image server with plugins that send and receive (unless it's had some kind of update I don't know about.)

OSCeleton puts out no image data, so there's nothing for syphon to send (to my understanding). At that, it's a terminal line app, and not really awesome in user friendliness for general audience.

Receiving positions this way right now is cool because - it's there, and it works. It's tremendously indirect... but thankfully, latency is good. OSC should not have to be part of this in any way, unless desired (from a pure design, eloquence of function standpoint.)

monobrau's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

I meant it more like besides the osc data, rerouting the depth and cam image from the sensor server as well trough syphon, so you could use them in qc and other syphon supported apps. (My knowledge is limited about the workings of the server/osceleton so I could be totally wrong here)

SteveElbows's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Yeah you could use Syphon for that. Openframeworks would probably be the easiest way to build such a thing, as I think they have that framework working with OpenNI etc, Syphon is supported, and OSC is trivial to do in openframeworks. Having said that you could also just write a QC plugin using openframeworks and not have to worry about the Syphon side of things at all.

The reason I remain interested in an OSC solution is because I have applications where I will want to control more than one computer, or more than one application simultaneously from a single sensor & skeleton data, and by using OSC I can keep quite a few options open in this regard.

SteveElbows's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Could really use a few more QC users voices on the discussion on the OSCeleton group, as right now its just me & George approaching the issue from completely different directions and so disagreeing quite a bit. Im bound to think that my 'keep it simple' approach is the right one but there is no point in me keep repeating myself, and I may be wrong, so could use some additional opinions!

http://groups.google.com/group/osceleton/browse_thread/thread/666ee7f423...

Cheers!

dust's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

sweet going to try this out i have open ni running on my mac still need to explore it a bit. osc seems like a good solution. cant wait till the unity wrapper is working on mac ;) will post results

gtoledo3's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

I think Amir may have gotten the Unity wrapper working yesterday, so you might want to check google groups. Might be wrong on that, I've only been following it superficially. I do think that he got over the main hurdle, once he figured out he needed to be installing libusb as universal (errr... this is what my foggy memory seems to have retained.)

dust's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

i had to uninstall libusb devel as i installed it with open kinect. also i built the sensor binaries for the mac from linux not sure if that makes a difference or not. going to build the osceleton and try this with quartz.

dust's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

don't know if this will help anybody get started but i posted my terminal print of how to install openNI on mac. here its pretty simple.

http://www.dustinoconnor.nmdprojects.net/index.php?option=com_content&vi...

this is as far as i have made it. i'm getting a segmentation error when i try and run osceleton ? any suggestions.

monobrau's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

It gives the same error on my setup, but it's completely usable though.

dust's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

your getting the same error but osceleton still works ? interesting.

i need to investigate some more but it seems its possible to just grab callbacks from the sensor server in c++ if thats the case i think i might be able to make a plugin for qc ?

either way it will have to wait i hosed my sudoers file file so mac ports is busted going to restore back to yesterday.

monobrau's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Probably so, I believe that's what OSCeleton does, forwarding data from the sensor server.

If it helps, this is the installation procedure I went trough: http://www.oneseconds.com/dev.html

itsthejayj's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Dame so close same problem 'Segmentation fault' on the osceleton-osx app! Any new ideas on this?

Right as for all your OSC string structure problems feel free to use our BBOSC send and receive plugin built for Just Add Music. However for the meantime it has a Creative Commons, Attribution - Non Commercial - Share Alike 3.0 license.

PreviewAttachmentSize
BBOSC.zip690.66 KB

gtoledo3's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Did you try the OpenNI User Tracker first... does that work?

gtoledo3's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Oh, and thanks for the OSC patch, it looks really nifty.

dust's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

love bbosc its a great framework. seems like a better route to build a cocoa latch to the sensor server and output in a plugin possibly.

the segmentation error bugs me, it either has something to do with the build being a different gcc maybe or a buffer overflow ? either way a few people seem to get it working.

dust's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

so after the inspection of the osceleton c++ file it seems apparent that i'm an idiot and didn't close the sample-skeleton-tracker i had open in order to test if nite was working.

i knew it was working as i was able to man -h osceleton-osx. well thats solved user error.

here is a print out (see attachment) of the unix commands and orders of operations if your not familiar with unix then this should help a bit. it is soley predicated on the fact that you have installed mac ports, x-code, and libusb-devel +universal.

hope this saves someone time as its not the directions are not entirely explicit although they are all there in the read me's.

the things in bold are the commands you need to do.

PreviewAttachmentSize
osceleton.rtf20.36 KB

itsthejayj's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Oh hang on sorry the plugin name is misleading, the BB stands for bestbefore the company i worked for that developed it. Its actually using the vvv framework.

I fixed the segmentation bug by closing all video inputs and reinstalling all the Open NI files

dust's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

oh i thought you where talking BB as in bbtouch ben brighton. yeah vvv is a good framework. you should change the class names around if you use it in a qc plugin though as lots of people use that framework to make plugs and it tends to give you errors when you make plugins with the same class names in qc. hope they fix up someday. i figured out the segmented error i was running the sample in the background.

idlefon's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Cheers for the plugin!

What does "OSC Label" and "Path" mean in the patch? I tried the listening port and the IP but didn't work with OpenTSPS. Also could you explaine the last two objects in the "setting" section.

Thanks.

SteveElbows's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

gtoledo3 wrote:
I think Amir may have gotten the Unity wrapper working yesterday, so you might want to check google groups. Might be wrong on that, I've only been following it superficially. I do think that he got over the main hurdle, once he figured out he needed to be installing libusb as universal (errr... this is what my foggy memory seems to have retained.)

When you wrote that Amir had made a breakthrough, but the Unity wrapper was some way from working fully. Its closer now, I tried it last night and I think it just needs some performance optimisations and a tidying up of the Unity scene so that things are wired up properly.

SteveElbows's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

Looks like the QC-compatible mode for OSC messages has now made it into OSCeleton, yay. Not tried it myself yet.

https://github.com/Sensebloom/OSCeleton

And drifting off-topic again, the alternative Unity wrapper for OpenNI is getting there but still has severe issues that make it no fun to use on the mac right now.

dust's picture
Re: OSCeleton with QC (Kinect skeleton tracking via OSC)

awesome. im using osceleton with unity. i would think that it would totally be possible to do some sort of native nite implementation in unity. the open ni dll "OpenNI.net.dll" gets installed into my mono framework on my mac which is what unity is built on.

still trying to sort out this middleware thing but i would think it will be totally possible soon to have a native implementation of nite on unity for mac. not sure what that requires, probably a pc which i don't have. until then you can use osceleton with this unity/osceleton package i made. it uses waxtastic's qc parser sp you can serve the skeleton to other apps at the same time as unity etc..

  1. make new unity project.
  2. import uni-osceleton package.
  3. run waxtastic's max-msp to qc app.
  4. run osceleton like code sample below.
./osceleton-osx -p 8110 -mx 20 -my -20 -mz -10 -ox -10 -oy 7.5 -oz 10 -a 127.0.0.1

this just places spheres at the respective joints. soon i will see about getting a skinned model rigged up.

PreviewAttachmentSize
uni-osceleton.unitypackage.zip17.87 KB