Kineme 3D Alpha 9Augmented Reality? Fun with AR and Single Marker Trackers...

gtoledo3's picture

I managed to realize that the AR Toolkit single tracker marker structure that was posted to the Apple QC Developer group, to work with .MQO (a windows type 3D file) could be "decoded" by mapping the appropriate structure to work with the stock 3D it could work to make any typical structure in QC "move".

I did this simply by waving a test sheet in front of the camera while observing each part of the structure, looking for plusses and minuses. Then I sorted the relevant part of the structure with the "structure index member". So far, I have only found I have had to scale the "rotations" with math, to get them to track right... up, down, left, right, worked pretty close right away.

To make it flashy, I used an Alpha 9 patch inside of the stock Apple 3D transform. But it looks like the AR Toolkit motion detection stuff could be mapped to any "number" input in QC.

jrs's picture
Are you able to post a macro

Are you able to post a macro patch or the like for this please? even just the scalings you've worked out

gtoledo3's picture
I 100% plan on

I 100% plan on posting a .qtz where the single marker tracker "matrix" output is "decoded" with the structure tools to interact with the Apple 3D transform patch. Probably tomorrow. The AR Tools based single marker tracker is something that was posted to the QC Developer mailing list, along side an MQO loader... which is actually a "Metasequoia" file... a popular Japanese freeware 3D modeling program, that only works on Windows. A couple of the renders I have on Vimeo use that...

But I grew frustrated, because I didn't even want to go down the route of working on anything where I would have to go back and forth between Windows and OS X. I really wanted to figure out a way of using the single marker tracker portion, with all of the stock Apple patches.

I think that is a really universal way of approaching it, because you can just put whatever you want into a 3D transform, and it will move along with the "tracker" print out. It is somewhat update proof in that way... you could take regular particle systems, any macro you already have, and just stuff it in the 3D transform. This opens things up quite a bit, and allowed me to use Kineme3D Alpha9 macros that I already had whipped up.

I will NOT be posting any Kineme 3D stuff/ .qtz's :o) I think that is up to Steve and Chris. I have just been given testing/rendering permission, and I am very grateful for that :o) I have done pretty much all of the renders with Quartz Crystal and Kineme3D Alpha9 on a 1 gig RAM Macbook, so it is safe to say that Steve and Chris are basically geniuses. I can't say enough good things about Kineme and Kosada in general. I only wish they they needed a full time "renderer", lol.

gtoledo3's picture
a couple more clips...

....and this following one uses the cool as "heck" subsurface scatter shader (GLSL) that toneburst posted not too long ago, on the moving Pagoda object... as well as the Kineme 3D Alpha 9. Quick and nasty screengrabs using iGrabit.

The plug ins were posted here... I have compiled versions if anyone can't get it going in X Code.

But again... I'm not using the MQO importer, other than my initial testing, because I am not crazy about the idea of installing Metasequoia on my PC, and having to go back and forth between that and the Mac (or even just having to mess around in Windows period). I also found the already made/ free MQO files to be pretty limited, with really unpredictable results. After the MQO plug crashed 6 times in a row, using files straight from Metasequoia, I felt like it wasn't worth wasting time with (not trying to be harsh, it is just unstable opening up this Windows file type on the Mac). I also had a weird problem with not all of the MQO files importing correctly... a model cut right down the half, one that would load once and then dissapear, the aforementioned crashes with company made MQO files. I also didn't like that MQO files seem to introduce potential for virus ( I could be wrong on that, but there are reports of MQO files being able to harbor "viruses").

However, it is worth messing with as is... it is just that the MQO part is really sketchy in my opinion. The MQO "office equipment" files that were provided all worked well.

I am still tweaking the math on getting the single marker tracker plug (the one I have been using), to work normally with other Apple QC patches. I said today, but more testing has revealed that I don't have some of the "centering" quite right at some points in rotation, so I want to work on getting it a little more accurate.

The basic concept is that I used the single marker and structure tools to sort through the output info of the single marker tracker, selecting the correct numbers of the structure to a 3D transform. Some of the "info" of the structure matrix required scaling (the rotatations in particular), so those had to be attached to math patches (and multiplied by about 100x... I recall). So if that makes sense to you, that would be your starting point.

If it doesn't make sense, don't worry too much, because I will definitely still post a .qtz that hooks up the single marker tracker to structure tools, the required math, and then a 3D transform.

DanieleCiabba's picture
I'm fighting with the same

I'm fighting with the same problem...

cwright's picture

mqo files appear to come in 2 flavors: "old" and "new". I've got an example loader for one of them (not sure which, comments are all in Japanese), but nothing in that sample seems to indicate the capability of storing viruses. The loader could misbehave on bogus input, in theory (though it looks like it handles errors rather completely), but that's a loader thing, not a characteristic of the file format itself.

(That said, it looks like a dreadfully lousy format, don't expect support for it in kineme3d anytime soon).

If you get a moment, could you fling a couple files my way (for inspection)? I might be mistaken, and there might be benefits to adding support for this as well (however unlikely it may be).

gtoledo3's picture
Chris, it's my

Chris, it's my pleasure...

I'm going out for a bit, but I'll send some sample qtz's before the night is out. One with just the example mqo setup, and then one/some with the Kineme3D.

Tighten up the 3D transform, or use some of your keen observational power (and actual programming know how, as opposed to my trial and error), to figure out how to do other cool stuff, and we will all be the better for it :o)

I'm extremely stoked about this, to no end. To have the "dinky" ( I feel like I am ripping on the MQO format, but I'm just not impressed) fighter airplanes working with AR is one thing, but this looks cool as all get out with the Kineme stuff.

The one thing that is somewhat interesting with the "quartz dev/ blog" mqo loader, is that you will see that it loads skins... but I actually think that might be part of what is causing it to crash. I also think that there are some kind of part of the Windows coding that isn't written properly in their loader. You will see how that airplane I have in the one Vimeo sample clip doesn't have all of the rear rudder. I notice that other files do that... only "half" of a part of the object will appear, instead of the whole object. Other objects load with "no" skins, even though they should actually have skins (I believe).

gtoledo3's picture
Done :o) Hopefully your

Done :o)

Hopefully your mailbox can take that!

gtoledo3's picture
Yeah, I have had MQO files

Yeah, I have had MQO files crash QC way too many times now, I am almost loathe to keep trying them. But some work great!

I sent Chris a copy of this, since most of it that I have done is using the Kineme3D and not the MQO loader stuff.

I would rather have Chris look at my decoding of the structure matrix and see if he has anything to add before posting it up here. Also, this tracker isn't even a Kineme plug-in, and I tend to think that a Kineme implementation might be a great deal better. Soooo... I am going to let it lay a little bit, and see what Chris's input is before posting anything.

cwright's picture

3.0mb? that's nothing :) got it, i'll take a look when I have some downtime (haha...)

gtoledo3's picture
I notice that the fella just

I notice that the fella just added a matrix to parameters... but it doesn't work right for me. It tracks for a second, and then screws up! I hate that! But I don't want to be down on it, because the effort is cool...

I'm posting this for anyone that compiles the single marker tracker (which is what I have been finding to be useful...)

gt-Single Marker Detection To 3D Transform.qtz4.77 KB

Scratchpole's picture
Hmmm? xcode 3.1.1 built it, but?

Ok so I got the single marker tracker to build in the latest version of xcode, but it is still red in the xcode project window, it say's it built successfully though. How do I get it working with that qtz example provide by gtoledo?

cwright's picture

depending on the project, it might have automatically copied the plugin where it needs to go (which would make it red in xcode, since the file's "gone" as far as xcode knows).

If not, you'll want to look in the project folder /build/Debug or Release/ and there you'll find a .plugin, which you should copy to your "~/Library/Graphics/Quartz Composer PlugIns/" folder (unless it's using the unofficial API, in which case that last part is "Quartz Composer Patches") -- it's probably the first though.

Check for any interesting messages if you continue to have problems, and feel free to post screenshots if you can't get it worked out.

Scratchpole's picture
CV is working, yipeedooo!

Thanks Christopher et al. Just had a few bits of the jigsaw missing. I think xcode did put it somewhere automatically but it was in the wrong place (Users/joecatchpole/Library/Graphics) and I guess that conflicted with~/Library/Graphics/ where I put it.

I brought the dynamic surface projection quad into the commented video tracking composition. Whoopee.

Now how do I get it to control a 3D object or particle system, think that's for later tonight. What I want is for a particle system to follow a dot around so I can have particles doing some handwriting. Any tips? Thanks.

Picture 1.png
Picture 1.png285.02 KB

gtoledo3's picture
One thing... I have noted

One thing...

I have noted that I am using the same structure member for x and z rotations... it "works" for some reason though! I know that is what is making my scenes "face" the camera even if I turn the paper... View my structure decode as a workaround and starting point... all I did was wave the piece of paper as I "hovered" over the structure node, and looked at the structure changes to try to figure out what is what.

I would definitely try the matrix to param plug in that the fella posted... it only works for me for a second, and then it seems to interfere with the tracker plug in somehow. Actually, just having it installed seems to cause abberant behavior... that could be only on my system, who knows.

I would try to look at the particle paint qtz for what you are describing, but I'm not sure what is the most logical way to hook it up using this AR thing...

DanieleCiabba's picture
that's something

that's something that could help you...?

gtoledo3's picture
Or maybe THIS might help you :o)

I have mocked up a real hack job of optical flow and particle paint together...

I would like to have sometime to get some of the "physical properties" interacting together... I have a few more optical flow type of particle compositions that I have been working on, mostly with "wind gust" by waving hands, type of concepts.

I posted another .qtz render on vimeo today, where I worked it so that the motion makes the particles "bloom" and get bigger for a few seconds after sensing motion....

But for this...

-Hit the trackpad bar, to actually enable particle paint. There are other ways of "enabling the paint function", like actually setting it up so that the macro only turns on when it sees motion... but I haven't fine tuned those.

-In order to turn the video "off", it is actually setup so that you are really turning on a clear that is "inside" of the particle paint macro.

-And truly, it would have made sense to put all of the optical flow part of the chain in it's own macro, so that it would all look more organized.

-You may see some unattached nodes, etc.... I would rather post this, and have it as a starting point for other people than to spend more time "tightening" it up.

You probably need- the Optical Flow plug-in (included in the zip), GL Tools, and Particle Tools. Install the Optical Flow in the regular plug in folder, as it is official API.

Let me know if you have any problems with this composition.... I "flattened it", and I also notice that it seems to respond much better in "bright" room lighting. As my living room dims at night, the tracking doesn't seem as good.

Also, try manipulating the friction and mass controls, and just using your hands/ finger tip... I have ALMOST got that working well be tweaking various parameters.

"Body Paint" QC Motion .002 , Particlepaint with Optical Flow testing from George Toledo on Vimeo.

Kineme Particlepaint with Optical Flow.zip20.95 KB

gtoledo3's picture
Optical Flow with Kineme Particles

This is a similar idea... but with a modified Voronoi type of pattern that I've used in a lot of my comps. It is set so that the particles "bloom" when they sense motion.... as well as x/y movement.

Turn on/off the "enable" switch in the lighting macro to enable "only" the particles and background, and no video feed background.

QC motion .001 from George Toledo on Vimeo.

optical flow with kineme particles-flattened.qtz23.32 KB

chiara's picture
add pattern to the single marker detector patch

Hello George,

Maybe, you're so kind to tell me please, how can I add more pattern to the patch "single marker detector"? This patch have by default just the pattern "Sample1 and sample2". I'm interested also in pattkanjii, hero and multiple pattern. I've try to add them opening single marker detector with dashcode, but some error occurred.

Thanks and many compliment, Chiara