opencv motion tracking

jacob360's picture

Hi everybody,

I was wondering if anyone could enlight me about motion tracking and the kineme opencv patchs.

I have to create projections for a theatre show and id like to use quartz for it.

I need to track the silhouette of actors and project onto them only and avoid the background.

Is it possible with the opencv patchs ? I have read other threads about it, but I couldnt find anything to help me regarding realtime drawing the shape of the actors and use it as a mask.

Thanks. Jacob.

usefuldesign.au's picture
Re: opencv motion tracking

jacob360 wrote:
Hi everybody, I need to track the silhouette of actors and project onto them only and avoid the background.

Big ask!

I would be looking to have a camera mounted in a similar position to the projector with similar lens length. Then I would see if I could pass the image through a threshold CI filter to create a feathered mask of darkest areas. Then use that mask for the projection.

How and what you are using as source for the projects could complicate matters significantly. If you are not trying to do any perspective correction or anything (like advertisements painted on sporting field surfaces for one particular camera view) with the projections this might not be toooo difficult.

The only reason for tracking the shadows would be if the projected image needs to track in addition to being masked. That would add another layer of complexity but I can't help with that part.

Good Luck

dust's picture
Re: opencv motion tracking

This seems to be a complex task. It would be fairly easy as stated above to threshold out the image of the actors and then blur and invert it to create a mask of the person. Then apply the mask to the projected footage. This will take care of the primarily 90 percent of e spillage from the projector maybe all depending on the stage backdrop. Your next step is then track the person so the visuals will follow them. This is the com plex part. You have basically two options good track points or haar or a combination of both. I like a combo but basically you will want to maybe just use 2 points to track when starting and put the points reload to update a cowpoke times a second this way you will always have a good track. Then simply use the point data as coordinates for the projection. The complex part is if there are two aft ors going in different directions. At this stage you will want to use the haar classifier as it supports multiple recognitions. Ie.. It can track two faces independentnof each other. To do this your going to need a set of haar classifiers for the whole body.

sbn..'s picture
Re: opencv motion tracking

Another option would be to go the hardware route. Many webcams are easily moddable into infrared only cameras. I've done it myself to a cheap Logitech, and the Playstation Eye is supposedly easy to do this to as well, and has decent third-party OSX drivers. Or so I hear.

In this case, you could take advantage of the fact that your projector probably doesn't output IR light. Then, you'd place an IR floodlamp near the cam (I've succesfully made a cheap IR filter for a stage spot out of red+blue standard filter sheets). The tricky part will be controlling the background and dancers' suits. Ideally, you'd have a IR-reflective background and IR-absorbing clothing. Do this right and you'll have a significantly cleaner input image than with visible light, regardless of other lighting / projections in the room.

Depending on the application, you can skip the tracking entirely and just transform+threshold the cam image to use as a mask.

usefuldesign.au's picture
Re: opencv motion tracking

Good point about the projection interfering with the mask. I didn't clock that.

Typically projectors (especially LCD video) are factors of ten less luminous than even small theatre lights so I doubt the projection would puncture a well calibrated threshold mask if surrounding illumination is good but there's only one way to find out.

jacob360's picture
Re: opencv motion tracking

thanks for your answers! its very useful. We thought of using cam+IRfilter and then use the threshold to get a live mask image. Apparently this would be the easier solution. As you said Dust, tracking with good points to get the projection move with the actors would be the next step.

I am trying to make it entirely with quartz, however another solution my friend developed was to use eyesweb - get the coordinate through pure data - send it to quartz. I searched a bit about eyesweb and it seems it can realtime draw the shape of the object. That is why I was wondering if we can do the same with the opencv (had no time to read extensively about opencv for the moment) and if this part was brought in quartz with the kineme opencv plugin. Am i wrong or can we do this kind of tracking with the opencv librairy ?

dust's picture
Re: opencv motion tracking

actually once you get a good mask and use the masked image for tracking you should be able to get a pretty good result.

i guess it all depends on what the actors are wearing and the backdrop. there are other ways to detect motion as well in qc like comparing the difference between two frames etc... you queue up an image and use the difference between the before and after to get a reference point of where the motion is. this i guess only works if people are moving.

there's a guy on this forum from france i believe that is doing the IR method with pingpong balls attached to the actors.

i think you be able to do do this all in qc. the one issue i see is if the actor leaves the screen, that is why i suggest reloading the track points every second.

leon's picture
Re: opencv motion tracking

You can reduce interference of the dancer suits placing the light source on another angle than the camera, like just in front, so the IR falls "behind" the performer and reflects from there. I guess... The example Dust refers to is http://kineme.net/forum/Discussion/DevelopingCompositions/Motiontracking...