Gesture Recognition Qtz Tutorial

dust's picture

So i know there has been lots of rants on how to create multi-touch gestures and what not a little while back. So i have been trying to port the viterbi algorithm to java script for a few months now so I can do advanced pattern recognition and other cool stuff, but for some reason I can not figure out how to do multi dimensional associative arrays with key in java script. Although now that i think about I might be able to do with kineme structure ?

So while I was tinkering around today with some multi - touch approaches while getting ready for a class I will be taking this semester on the related subject I came up with a great idea. That is to do pattern recognition or 1 to 1 point mapping to create the gestures I need with noting more than grade school math. The logic is a bit complex and there lots of work to be done, but within a few hours I was able to wip up a pattern matching system. No this isn't that fancy $ stuff or the wii-mote probability smoothing algorithms discussed on here. Im not sure if this would count as filling a feature request but it demonstrates how to record gestures and match them on the fly.

You can watch the video tutorial and get the source here.

I have been having some memory management issues with my server tonight but I think I have got some snappier settings. Im attaching the source here incase my site goes down. Last night I looked at my stats and I had 300 guests on my server doing what I don't know because my content hits are no where near that many. I guess people are watching my illusion on the homepage, I don't have that archived as content yet but just incase look at the attachment.

The 1 key records your gesture the right mouse clears the gesture recorded and the left mouse acts as single touch and triggers the runtime evaluation so the patch knows if a gesture as been hit or not.

I have left a few simple things up to someone to do. 1 add conditional and counter the more hits the better. 1 hit = false positive but 2 hits with my tolerance settings = gesture. i urge you to watch the video so you will see how it works but it dosent matter mess around with it improve it, laugh at I don't care I hope someone will find it useful. Like I said I accomplished more with a simple approach in an hour than I have done all year trying to convert viterbi.

so 1 key saves gesture right mouse clears left mouse activates. oh yeah you have to hold the one key down until your done with your gesture. eventually that will be a three finger trigger to swipe or something to enable recording i don't know haven't got that far yet.

just so you understand you can record a star gesture then set a tolerance. you can then draw the same gesture anywhere on the screen to get matches. This doesn't match big star with little star although some simple math averages could make that possible. And yes I know i could do this with a few lines of java script. Give me a break I like noodles. The macros can do the same thing even faster sometimes with simple things like this.

dust_gesture.qtz17.34 KB

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

dust's picture
Re: Gesture Recognition Qtz Tutorial

rant rant blah blah man i got to put quartz composer down after 2 am. im looking at this this morning and well for some reason i negated to reverse my structures coming out of the que but for some reason it still works its a bit wonky. it doesn't work particularly well its a tad slow but i was able to match a star configuration this morning. im really hoping there is someone out there that can see the faults in this and show me a better way. im going to try and do a different version today maybe more advanced and use the value historian patch maybe look into some of the feature request links and see if there is not something entirely to difficult to port.

gtoledo3's picture
Re: Gesture Recognition Qtz Tutorial

I looked at it for about 10 seconds and noticed that there were math patches with resulting values not attached, etc.

I'm not sure how/why this is supposed to work. It wasn't really making sense to me, but then again, I'm still in coffee mode.

dust's picture
Re: Gesture Recognition Qtz Tutorial

i didn't get to clean up my dangling math patches and stuff. basically im taking the sum of xpos and ypos then saving them into a structure or array. that is the saving part. then i have a left click boolean to start filling up another cue of xpos and ypos sums. the rest is just iterating through those structures and trying to get conditional hit matches based on a tolerance. i ran into trouble trying to do runtime evaluation of my gesture thats why this is a bit wonk. if you want to actually see this working the way it should or the principle its based on take out the runtime component.

to do this without a continuous evaluation. get your keyboard patch out. set number one for record gesture number 2 record gesture to match, number 3 evaluate matches, number 4 to clear.

so put a sprite in the middle of the screen to trace. ok when you hit the 1 key it trace the sprite and it will fill up ques with all your coordinates based on index, then reverse those cues for both x an y.

now do the same but this time use the 2 key to fill additional cues.

once you have both sets of coordinates recorder into a que. set the 3 key to index through your structures looking for a tolerable match. its hard to explain in qc terms.

here more cs terms or pseudo code.

gesture_xpos[100]; gesture_ypos[100];

gesture_ypos_match[100]; gesture_xpos_match[100];

tolerance = .05; threshold =0; index = 0; pos = 0; neg = 0;

g_pattern[100]; g_match[100];

while (index != 100) { g_pattern[index] = gesture_xpos[index] + gesture_ypos[index]; g_match[index] = gesture_xpos_match[index] + gesture_ypos_match[index]; index++ }

for(index=0;index<=100;index++) { if(g_pattern[index] > g_match[index]) { threshold = g_pattern[index] - g_match[index]; } else if(g_pattern[index] < g_match[index]) { threshold = g_match[index] - g_pattern[index]; } else {do nothing;}

if(threshold >= threshold - tolerance || threshold <= threshold + tolerance) { pos++ } else {neg++}


don't know if that makes more since.

i m just matching arrays. it would work the same if you sum the total of each array and compare totals instead of index as well. at least that pseudo code is what im trying to do all in qc now that i look at the patch it is doing something different but still working which is messing with me.

need to brew some coffe and wake this has been bugging me all night i couldn't sleep very well without having this ironed out. i don't know how im going to pass a multi touch class if i cant even make my own gestures.

cybero's picture
Re: Gesture Recognition Qtz Tutorial


You've got something interesting going on here, dust.

Haven't had a chance to look at your tutorial, dust, but shall drop by si02, some time after the weekend. [still shed loads of paperwork to do].

Might not be a bad idea to clear any composition posted of all unnecessary patches, newbies might even fret about deleting what isn't absolutely essential.

Of course if such unhooked patches are potentially useful when amending the composition for a somewhat more sophisticated purpose, those patches unused in the posted example are best annotated by purpose for your 'tutees'.

dust's picture
Re: Gesture Recognition Qtz Tutorial

yeah im not good with annotating or well even putting my name on things, sooner or later once i have a plugin that kineme hasn't thought of yet maybe i will start marking things up. i figure if i make enough video tuts then i won't need to mark up or explain many things in annotation. although i do have to do annotations for java i guess its a new feature. hey im just glad one person views or downloads anything i make. im an artist all i need is a constituency or audience of 1 to validate my work as art. otherwise it doesn't exists and well doesn't count if im the only one to view or at least thats what the art doctors tell me. this gesture thing isn't art though its just a tool to make the art later on. this is kind of ugly, i think code can be art as well or well aesthetic and elegant, more often than not my code is ugly, i think. when i turn in home work assignments thats when i turn in a book when the assignment calls for one page of code but im learning. im hoping this semester i can at leaste use QC for the multi touch class, not sure what the grid lab is using but we are making some giant networked super computer thing for biology departments to share info or something. maybe QC can be the render end for os x user i don't know but i want to incorporate it into my studies somehow.