openCV: "locking" images to edges

jersmi's picture

I thought I would start a new post regarding the user bonoitlahoz's recent post using openCV to determine points to attach movie clips, here: http://kineme.net/forum/Discussion/DevelopingCompositions/CVToolsDetecti...

I am using benoitlahoz's comp as a starting point to approach one of those impressive tracking scenarios I see in zuga's work and many others -- I'd like to attach iterations of an image to the detected edge, then have the images behave as if attached to the edge as it grows, shrinks, travels, etc. I included my test movie with this comp. Any advice is appreciated.

PreviewAttachmentSize
edge_detection.zip658.82 KB

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

jersmi's picture
Re: openCV: "locking" images to edges

Another question regarding this comp: how can I make it so the images are always on the outside of the edge, instead of some inside, some outside?

benoitlahoz's picture
Re: openCV: "locking" images to edges

Hi jersmi,

I'm very interested too by your first and second questions.

For the second one, I guess it depends of which are considered the first X and Y points in a line. What I would like to understand is if there is a relationship between these coordinates and another state of the tracked object, so we can test this parameter to know if the line is "well oriented" or not.

jersmi's picture
Re: openCV: "locking" images to edges

I understand what you are saying about testing tracked object states. One issue regarding the openCV hough lines patch is that it continually regenerates lines as tracking states change. Wondering if there needs to be another method of generating lines (or points) that, once generated, move with the tracked object.

jersmi's picture
Re: openCV: "locking" images to edges

Looks like going back to the OpenCV Good Points To Track (with Calc Optical Flow Pyr LK and the Canny patch) to generate a point structure is a decent way to "attach" points to an edge. However I can't see how to get useable line data for tangent / angle because successive points are not next to each other.

Maybe OpenCL is a good option for speed / accuracy, but that's down the road for me. (dust, if you come up with anything, please, please!)

jersmi's picture
Re: openCV: "locking" images to edges

Some progress. Using the point structure I thought I would try to simply get all images to point to the center (0,0) then decide later how to make them point to other coordinates, averages, etc. Using pX from pY (indices 2 and 3) with atan2(x,y) I get an angle but now have to sort for the 4 quadrants in QC units (x,y; -x,y; x,-y; -x,-y). I'll post when it's working.

jersmi's picture
Re: openCV: "locking" images to edges

Cobbled together more math to use the OpenCV point structure to "attach" images to an edge. This does not use the (very nice) line structure setup of benoitlahoz's -- only uses Good Points to Track with Optical Flow data, using (0,0) as reference point for image angle. This uses the same "clean" test reference movie as the first post. It's a start...

PreviewAttachmentSize
edge_detect001.qtz681.15 KB

jersmi's picture
Re: openCV: "locking" images to edges

Update with ability to follow mouse click (and drag).

I'd like to generate axes for live tracking "blobs" and have images point towards axis. Any ideas for this math would be appreciated.

(Not sure why this comp is so big...) (Sorry, I should wait to post so I don't have such a monologue, but I keep thinking I have to stop working on this, then I make a little more progress).

PreviewAttachmentSize
edge_detect001.qtz683.77 KB

dust's picture
Re: openCV: "locking" images to edges

edge detection is interesting. a few years back i made a max msp patch with cv jitt that found edges and tracked them to calculate the distance. i then attached a metronome kind of sound. kind of like how a garbage truck or a lexus makes sounds when backing up. i used the distance as an input to the metronome so when you got closer to an edge the sound would go faster.

the idea was to blindfold people and send them through a maze like installation and see if they could navigate through the maze without being able to see. that was a long time ago but these patches make me think about the patch.

there is a doctor on campus who is blind doing some experiments like this. he runs the spatial immersive and cognitive recognition / virtual reality lab. one of his students made a vr tetris game with sound so if you can't see you could still play. it was made with vizard a python based vr program. normally they do cognitive types of experiments but was interesting to play tetris with those crazy vr googles. kind of strange a blind man heads up the virtual reality stuff but im thinking of taking his human interface design class.

jersmi's picture
Re: openCV: "locking" images to edges

That's cool, a blind man doing VR!

And yes, anything you can pull out of the archives from your jitter experiments would be welcome, to apply to QC here and now.

benoitlahoz's picture
Re: openCV: "locking" images to edges

That's really cool !

I didn't take time to have a real look to the comp but playing with it is a pleasure. Your point tracking works very well. What would be great would be to make the triangles change their orientation with a lateral move of the lung/blob. I'm still thinking about my shadows...

@dust : I'm not blind but your max patch seems very very interesting ! :-)