Find location of uniquely colored object (Composition by jstrecker)

Author: jstrecker
License: (unknown)
Date: 2010.11.22
Compatibility: 10.5, 10.6
Categories:
Required plugins:
(none)

An implementation of Core Image functions from GPU Gems 3.

Adapted by @jstrecker and @smokris.


3rd-party plugins needed:

Usage:

  1. Get a colored object to track, such as a marker.

  2. Calibrate the color: Click Input Parameters, click Color, click the color picker (magnifying glass) in the color dialog. Hold up the object to the video camera and click on it. Adjust the Threshold until the object and nothing else is highlighted (as much as possible).

The white circle should follow the object.

Known issues:

  • The framerate is slow (on my machine, 8-10 FPS for a 256x256 image). Any suggestions welcome.

  • It needs square images with height & width a power of 4. (The input image is automatically adjusted.) With extra math you could get rid of this restriction.

PreviewAttachmentSize
color region centroid 0_1.qtz14.93 KB

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

cwright's picture
Re: Find location of uniquely colored object (Composition by ...

"Reduce To One Pixel" is a fancy way of saying "Area Average". It gives identical results, without needing JS to coordinate the decimation passes.

Curious that IPS gives different color values from Image Pixel... I wonder what's going on with that...

dust's picture
Re: Find location of uniquely colored object (Composition by ...

this works pretty nice with the kinect. here is a simple cl kernel that takes an average of the depth image and turns a certain depth green so you track with this centroid tracker.

__kernel void test_rgbaFFFF(__rd image2d_t srcimg, float x, __wr image2d_t dstimg, __global float4 *cur)
{
 
   int2   pos = (int2)(get_global_id(0), get_global_id(1));
   int tid = get_global_id(0) + get_global_id(1) * get_global_size(0);
   float4   color = read_imagef(srcimg, CLK_ADDRESS_CLAMP_TO_EDGE | CLK_FILTER_NEAREST, pos);
   cur[tid] = (float4)(pos.x,pos.y,((color.x+color.y+color.z)/3),0.0);
   if(((color.x+color.y+color.z)/3)<.1)
   {
   color.x=0.0;
   color.y=1.0;
   color.z=0.0;
   }
   write_imagef(dstimg, pos, color);
}
PreviewAttachmentSize
kinectcolortracking.png
kinectcolortracking.png1.06 MB

gtoledo3's picture
Re: Find location of uniquely colored object (Composition by ...

On my machine, the fps is exceedingly fast. I can do multiple centroid calcs and still get 60+fps out of it. This is when processing a 256x256 image, and using the nVidia 9600 (eg., the macbook pro 9400/9600 gpu version in "best graphics" mode.

I've had some different thoughts about id'ing the blobs, and even setup something, but I was wondering if any thought had been given to that from your end. After going through all of the CI code involved, what do you think about returning multiple centroids, and then, comparing frames between so that you can ID them?

I rigged up something using a hit test type grid to get it working, but it didn't feel too robust. It felt like I was adding an unneeded layer, and that there is a way to get multiple centroids from the CI calc that's eluding me.

dust's picture
Re: Find location of uniquely colored object (Composition by ...

i was wondering this as well george. i have been using open cv in ofx to get multiple blob centroids. this works great to get an over all centroid of a given color. i'm using it with kinect and have been able to the whole hit test working and tracking multiple colors.

so lets say you have two hands one is slightly behind the other then i'm able to assign a different color for lets say said left hand. obviously your right hand at some point is going to be at both depths. so follow me for a second. if depth x is set to color red and depth y is set to color green and if color green is a depth closer to the cam then at a certain point you have to travel through depth x to get to depth y.

so at this point your second red object will be at where your green object is because you have to travel through x to get to y depth. so you need to hit test to see if the red object is in the green space and if it is make sure its not enabled. once the y object is past the x depth you can add your second hand to x depth and you can track two centroids blobs for both hands.

however i to think this isn't a desired workaround as i would like to track centroids at the same depth. so cropping the screen to left side and right side and then tracking different colors for each side defiantly works and can work in a game controller context. as well as tracking different depths at different color can also let you track lets say your hand and elbow and body etc.. which is also great.

but like george i to am picking apart this great patch and searching for a way to track multiple of the same color masks like multi-touch. i'm thinking through iteration of the blob mask it might be possible to get centroid for each individual as opposed to the overall centroid.

either way this is a great patch it works much better than the the other color tracker ci kernel floating around.

gtoledo3's picture
Re: Find location of uniquely colored object (Composition by ...

If you think about it, each thresholded area has a different overall color value after it's been run through the coordinate mask. So, by comparing color values between two frames (at least), you can start assigning id's to them without relying on depth.

I guess it's like.. it seems like something needs to detect the areas of alpha do make it reduce to more than one pixel. A pixel for every blob detected... final extrapolation to coordinates needs to happen for every pixel detected... and it all needs to be dynamic.

I guess if instead of area average, everytime alpha occurs between blobs, you get a new pixel output (err, multi area average with areas dictated by alpha between), then you could detect the pixel count, and have it all work in an iterator.

... just thinking it out...

dust's picture
Re: Find location of uniquely colored object (Composition by ...

i was thinking of trying to do this all in a kernel by going pixel by pixel to find the contours. i guess sort of looping through u and v. sort of scanning for a black pixel followed by white pixel then looking for the next white pixel followed by black pixel on the same line and storing distance between them as well as the start and end contour points. then go line by line until you get a closed contour then fill any holes that may be in that blob and use the distance info to create a centroid.

i found that cvtools does a good job of tracking as well once you fill in the missing alpha of the depth mask. meaning with cv tools you can track two hands its not as smooth as this color tracker. with cvtools you need to blur out your tracking blobs to get rid of jitter.

have you tried cvtools find edge or lines yet. its pretty interesting if you do a peace sign for instance you can place a photo easily on the center or start or end of the line. so not tracking blobs but line centers is also another approach.

jstrecker's picture
Re: Find location of uniquely colored object (Composition by ...

I hadn't thought much about multiple blobs of the same color, but yeah, you might be able to adapt this composition to do it if you could partition the image into areas containing different blobs. Or it might be easier to start with one of the OpenCV blob tracking libraries, like cvblob.

@gtoledo3- thanks for the fps numbers. I'm on a MacBook Pro with Nvidia GeForce 8600M.

dust's picture
Re: Find location of uniquely colored object (Composition by ...

frame rate is not an issue for me either. here is a game i made for fun using this centroid tracker today. not much concept dribble basketball and throw basket ball but it only took a few hours to set up. i'm happy with the results.

Subtiv's picture
Re: Find location of uniquely colored object (Composition by ...

How can i transform another "layer" in the depth map to be transformed into green? I've been trying over and over but i'm not finding any solution.

Thanks!

dust's picture
Re: Find location of uniquely colored object (Composition by ...

The kinect patch depth image has been inverted so to get this kernel to work so you can track green for let's say your hand you need to invert the depth image before running through this or alternatively you switch the if statement from <0.1 to if >0.9. if you want to track depth layers and replace with a color just copy the kernel set to .7 or .8 to lets say turn your body green. Its very sensitive even if you can not tell by the depth image. Like .85 will give you elbows if your hand are in front of you etc...

jersmi's picture
Re: Find location of uniquely colored object (Composition by ...

This worked just fine for me when it was first posted, have not touched it since, but I cannot get it to work now. I posted a test movie, and there is a sprite moving in a circle in the comp for testing as well -- two simple tests! Any help is greatly appreciated.

PreviewAttachmentSize
color region centroid test.qtz23.14 KB
TrackingPointTest.mov1.41 MB

offonoll's picture
Re: Find location of uniquely colored object (Composition by ...

I don't think the composition is correct/finished, after 'Reduce to Single Pixel' doesn't make much sense to me. I am very interested as well how really should work this.

jstrecker's picture
Re: Find location of uniquely colored object (Composition by ...

@jersmi - Does the original composition (color region centroid 0_1.qtz) still work for you? The composition you posted is just a little different and it doesn't work for me either. You could try going to File -> Compare Compositions to see what's different, and either change yours until it works or the original until it breaks.

dust's picture
Re: Find location of uniquely colored object (Composition by ...

this centroid patch works good for me. here is a macro implementation, you got to remember to use the the transformed image output if you want to do any sort of augmentation as that is the re-sized image used for tracking so if you don't overlay the right image things may seem off ?

the reduce to a single pixel is how this tracks so smooth. from what i understand it is taking a region of interest by using the four corner points and derives the center point of interest and reduces that area down to one center pixel that is then smoothly tracked.

things like open cv etc give are very jittery and usually require a smoothing function to get a good clean track. this patch doesn't require this because of the pixel reduction.

PreviewAttachmentSize
qcblobCentroid.qtz32.14 KB

jersmi's picture
Re: Find location of uniquely colored object (Composition by ...

@jstrecker -- thanks for the response. the original comp does not work for me now (downloaded again just now and tried as is). after i set color / threshold for a uniquely colored object i am holding up to my laptop's camera, the tracking sphere moves erratically on a diagonal line along bottom left of viewer to top right.

btw -- MBP, NVidia 9400/9600 GT

one thing different (all i can think of): at some point probably after i updated the Image PixelS plugin to Image PixelS2. However, reverted to previous version does not fix.

What could it be?

gtoledo3's picture
Re: Find location of uniquely colored object (Composition by ...

Try testing with something other than iSight (?).

See if you make a Render In Image with a a small sprite inside (and clear), and make the sprite the exact same color as the search color, if stuff tracks as expected.

Preprocessing can help, depending on the color of the object. If an object is really reflective, it can seem like there's a great deal of variation in color with lighting and make the color tracker seem not so good. Doing median or mean filtering, gaussian, etc., can help even out the colors of the source.

If it used to work really great, and now it just doesn't, I'm not sure...

jersmi's picture
Re: Find location of uniquely colored object (Composition by ...

Thanks, @dust, for the demo -- good to see that. GT, I set off using a PS3 eye, as you requested.

Then~

Appears that on installing the new Image PixelS2, the plugin reverted to the type "RGB (luminance)" setting for any comps using the plugin on my machine. I changed the three in the comp to R, G and B, and now it has come back to life, though there is an offset in the tracking of the objects, in both the original posted comp and Dust's demo. Still, phew! Driving me crazy, this one! I'm looking in to the offset now. Thanks, folks! (Does it make sense to anyone that this could happen?)

jersmi's picture
Re: Find location of uniquely colored object (Composition by ...

I am experiencing now an offset of the tracking point, which was not present when I first looked at this comp.

Question: what were the original Image PixelS settings? I am assuming R/G/B, but now some adjustment needs to be made and I don't quite understand what needs to happen...

jstrecker's picture
Re: Find location of uniquely colored object (Composition by ...

Yes, the original PixelS were (top to bottom) R, G, B.

jersmi's picture
Re: Find location of uniquely colored object (Composition by ...

Thank you. Almost a double post, sorry but I'd really like this issue resolved -- I posted the comp with notes on the offset in the thread by the creator of Image PixelS2: http://kineme.net/forum/Discussion/Programming/ImagePixelS2

Relevant for one because I changed the math expressions in the posted comp to be accurate on my machine (with original Image PixelS, isight and uniquely colored object). Not sure why I needed to change this as I believe it worked accurately before.

Advice welcome on the offset issue for Image PixelS2 as well as your mention of the transformation necessary for different aspect ratios for accurate tracking.

jersmi's picture
Re: Find location of uniquely colored object (Composition by ...

Sorry more questions, trying to understand:

  1. might I ask how the math expressions were derived for x,y positions?

  2. In the comp you mention that Image Pixel should be equivalent to the Image PixelS plugin. Mr. Oostrik's website says his plugin is based on the optical flow downloader patch: http://www.magdatt.nl/software.html. So I am not sure how the built-in Image Pixel would be expected to achieve this?

gtoledo3's picture
Re: Find location of uniquely colored object (Composition by ...

This shows how to use area average origin and extent to divvy up screenspace into 4 different regions. When one triggers a color match, one will trigger one of 4 differently colored spheres, and track in that quarter of the image region.

PreviewAttachmentSize
color region centroid 0_ 4 Regions.qtz25.43 KB