Sweeping objects with body movement

protarco's picture

Hi There!

I'm trying to wipe objects from the screen using body movement. The idea is that the people can sweep leaves from the screen moving their arms, so the objects can fly with a physical behavior revealing the image in the background.

My approach until now has been to isolate the person's figure adjusting contrast, and then creating an image that is only black or white, like a mask. Then my idea is to use this mask to repel the leave's positions.

I've tried the CV objects to do this but seems like there is no object for isolating blobs? face detecting works great, also point structures... but I couldn't find a way to detect a whole body in the CV tools.

I've also tried vade's optical flow and other objects with no better result...

Is there a way to build a particle cloud in a way that i can distribute its particle's positions in space using body movement? Is it possible to do this using a structure?

I've been able to distribute the objects of a structure in space but I don't know how to control each object's movement.

I think I'm gonna end controlling each object individually, in this case, anybody knows a repel algorithm that I can use?

Any help will be useful,

thanks!!

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Raconteur's picture
Re: Sweeping objects with body movement

Hey protarco,

Try this out. It is based on CVTools-0.2, and uses an image for the interacted with object. You could easily replace the fly with a leaf.

Click the screen to reload the images.

HTH,

Chris

PreviewAttachmentSize
Flies!.zip0 bytes

protarco's picture
Re: Sweeping objects with body movement

Hi Reconteur!

thanks for your answer

for some reason i cannot download your file, it says the file doesn't exist.

Could you please try to upload it again? try not to use "!" in the file name

thanks!

Raconteur's picture
Re: Sweeping objects with body movement

Sure. Sorry about that...

PreviewAttachmentSize
Flies.zip182.29 KB

Raconteur's picture
Re: Sweeping objects with body movement

My fault... I posted before hitting Attach. See the 0 bytes in that first file?

80

protarco's picture
Re: Sweeping objects with body movement

Cool,

thanks!

I like your composition very much!

but I think I'm trying to achieve something different. Imagine a screen full of leaves, and when a person comes into the scene, the leaves fly away.

I think this could be done by creating a structure grid of leaves and controlling their position in a way that they repel the person's shape...

Sorry, my english is not very good and I'm trying to achieve something very specific...

thanks!!!

toneburst's picture
Re: Sweeping objects with body movement

I'm guessing the problem would be getting usable data from the video. I've not really looked into this much, I must admit, but my guess would be that you'd probably have to do quite a lot of post-processing on the image, and/or light very carefully. On the other hand, if you're making an environment for the piece to be presented in, you stand a much better chance of getting it working.

a|x

gtoledo3's picture
Re: Sweeping objects with body movement

You should look at the Optical Flow plugin from the 10.5 Developer Tools (and probably on the Apple website), and the two associated example qtz's.

protarco's picture
Re: Sweeping objects with body movement

Yeah, I know what you mean.

The set is like this:

a video camera from the surveillance industry that gives me PAL signal in B/W. It has IR light built in, enough to cover the space in front of the projection. I have isolated the IR spectrum by adding a IR filter to the camera, so no light other that the IR from my camera is interfering. With some luminance and contrast adjustment I get a pretty clean image.

I guess the question is: is there a way to create 'avoidance behavior' in an object movement into QC? How can I say an object to avoid a particular position?

I guess this is what I'm trying to achieve:

http://www.youtube.com/watch?v=FGXBy5dg0HY

objects being swept by the body movement.

I think this is one big missing of openCV for QC. But I'm probably missing something or doing something wrong, or misunderstanding the cvTools, or not finding the right object, or just not doing things right ;-)

dust's picture
Re: Sweeping objects with body movement

echo what George said. there was some discussion a while ago in regards to body movement as painting source. which both george and i have been experimenting with. if you check the repository i have an example that uses body movements to splatter paint on the screen which will cover both the optical flow method and the particle method your referring to. if you like how the face detection works in cv tools then just download open cv and take one of the haar classifiers for a full body and replace so it will track the body instead of the head. or track the head the body the eyes all together etc.. there is file over at nuigroup that is a haar classifier for hand detection that might help as well. your other options may be some of apple ci kernel sources for green chroma keying and color tracking. as well as vade has made a ofx2qc plug-in where you can integrate open cv blob tracking from ofx into a qc plugin.

protarco's picture
Re: Sweeping objects with body movement

Thanks!

you just placed me in the right direction!

I've being playing with your splatter painting composition Dust, it's fantastic! I'd like to experiment with full body recognition using Haar cascades, but I'm having trouble finding one of those

There are other examples you are mentioning that I'd like to try like the vade's ofx2qc plugin, but it is not easy to find,

Any idea where can I find these files?

thank you so much!

protarco's picture
Re: Sweeping objects with body movement

Ok, I've stopped being lazy and did my research ;-)

I found the example files that George mentioned for the optical flow at OS X 10.5.

that's exactly what I was looking for!

I couldn't find a Haar classifier for detecting blobs neither full body... but I'll still looking

thanks again!!

dust's picture
Re: Sweeping objects with body movement

classifiers are a bitch to make they work pretty good unless you funny looking and don't look anything like 10 thousand images you need to make them with. like when i had big funny looking glasses like the ones in my avatar the face detection didn't work on me so well. i made 3rd person game last year in unity that was entirely controlled by facial recog. this year i adapted the technology to strapping a projector and wii mote to my head for some more degrees of freedom. here are some haar classifiers. choose what body part you want to pick up leaves withs.

PreviewAttachmentSize
haarcascades.zip2.03 MB

toneburst's picture
Re: Sweeping objects with body movement

Have you seen this, by the way?

http://memo.tv/body_paint

It's currently on display at the Decode exhibition at the V&A here in London.

a|x

gtoledo3's picture
Re: Sweeping objects with body movement

The first exhibit I saw that did motion tracking in a real cool artistic way was a thing where you would walk down this hall at an airport, and leaves would kind of follow you, as you made inferred wind. Wish I could remember more specifics...

dust's picture
Re: Sweeping objects with body movement

i saw something similar in an airport that wasn't leaves but it was like a spot light that would follow you. it was funny, people would try and get away from it would keep following them.

protarco's picture
Re: Sweeping objects with body movement

Yeah,

Dust, thank you for the haarcascades! great source for tracking body!

For this particular case, I've decided to use optical flow instead.

By the way, I've seen Memo's body paint piece, I think it's amazing. I'm currently in Berlin, so I think I'll miss it live, but what a great work.

Here is what I've been able to achieve until now, I hope you like it.

I'm now trying to give physical behavior to each object so they seem to fly when people'll pass by...

Hope you like it, thank you very much for your support!!

PreviewAttachmentSize
Leaves.qtz56.87 KB

ericjcart's picture
Re: Sweeping objects with body movement

Is there any way you could possibly post a video of what you came up with? I'm trying to achieve the same kind of effect and I'm really interested in seeing what you've done.

(I don't currently have the plugins required to view your qtz file.)

Thanks!

Raconteur's picture
Re: Sweeping objects with body movement

My abilities with QC are severely limited by my ignorance of 3D math beyond a basic level.

Can anyone point me in the right direction for some reading material (books or web sites) to help me understand the hows and whys of things like the JS functions attached to the CI Optical Flow node protarco's Leaves.qtz comp?

From a programming standpoint, it makes sense, I just don't know WHY or HOW that stuff was arrived at.

Thanks for any guidance.

C