Find location of uniquely colored object (Composition by jstrecker)

"Reduce To One Pixel" is a fancy way of saying "Area Average". It gives identical results, without needing JS to coordinate the decimation passes.

Curious that IPS gives different color values from Image Pixel... I wonder what's going on with that...

this works pretty nice with the kinect. here is a simple cl kernel that takes an average of the depth image and turns a certain depth green so you track with this centroid tracker.

__kernel void test_rgbaFFFF(__rd image2d_t srcimg, float x, __wr image2d_t dstimg, __global float4 *cur)
{
 
   int2   pos = (int2)(get_global_id(0), get_global_id(1));
   int tid = get_global_id(0) + get_global_id(1) * get_global_size(0);
   float4   color = read_imagef(srcimg, CLK_ADDRESS_CLAMP_TO_EDGE | CLK_FILTER_NEAREST, pos);
   cur[tid] = (float4)(pos.x,pos.y,((color.x+color.y+color.z)/3),0.0);
   if(((color.x+color.y+color.z)/3)<.1)
   {
   color.x=0.0;
   color.y=1.0;
   color.z=0.0;
   }
   write_imagef(dstimg, pos, color);
}

Preview	Attachment	Size
	kinectcolortracking.png	1.06 MB

On my machine, the fps is exceedingly fast. I can do multiple centroid calcs and still get 60+fps out of it. This is when processing a 256x256 image, and using the nVidia 9600 (eg., the macbook pro 9400/9600 gpu version in "best graphics" mode.

I've had some different thoughts about id'ing the blobs, and even setup something, but I was wondering if any thought had been given to that from your end. After going through all of the CI code involved, what do you think about returning multiple centroids, and then, comparing frames between so that you can ID them?

I rigged up something using a hit test type grid to get it working, but it didn't feel too robust. It felt like I was adding an unneeded layer, and that there is a way to get multiple centroids from the CI calc that's eluding me.

i was wondering this as well george. i have been using open cv in ofx to get multiple blob centroids. this works great to get an over all centroid of a given color. i'm using it with kinect and have been able to the whole hit test working and tracking multiple colors.

so lets say you have two hands one is slightly behind the other then i'm able to assign a different color for lets say said left hand. obviously your right hand at some point is going to be at both depths. so follow me for a second. if depth x is set to color red and depth y is set to color green and if color green is a depth closer to the cam then at a certain point you have to travel through depth x to get to depth y.

so at this point your second red object will be at where your green object is because you have to travel through x to get to y depth. so you need to hit test to see if the red object is in the green space and if it is make sure its not enabled. once the y object is past the x depth you can add your second hand to x depth and you can track two centroids blobs for both hands.

however i to think this isn't a desired workaround as i would like to track centroids at the same depth. so cropping the screen to left side and right side and then tracking different colors for each side defiantly works and can work in a game controller context. as well as tracking different depths at different color can also let you track lets say your hand and elbow and body etc.. which is also great.

but like george i to am picking apart this great patch and searching for a way to track multiple of the same color masks like multi-touch. i'm thinking through iteration of the blob mask it might be possible to get centroid for each individual as opposed to the overall centroid.

either way this is a great patch it works much better than the the other color tracker ci kernel floating around.

If you think about it, each thresholded area has a different overall color value after it's been run through the coordinate mask. So, by comparing color values between two frames (at least), you can start assigning id's to them without relying on depth.

I guess it's like.. it seems like something needs to detect the areas of alpha do make it reduce to more than one pixel. A pixel for every blob detected... final extrapolation to coordinates needs to happen for every pixel detected... and it all needs to be dynamic.

I guess if instead of area average, everytime alpha occurs between blobs, you get a new pixel output (err, multi area average with areas dictated by alpha between), then you could detect the pixel count, and have it all work in an iterator.

... just thinking it out...

i was thinking of trying to do this all in a kernel by going pixel by pixel to find the contours. i guess sort of looping through u and v. sort of scanning for a black pixel followed by white pixel then looking for the next white pixel followed by black pixel on the same line and storing distance between them as well as the start and end contour points. then go line by line until you get a closed contour then fill any holes that may be in that blob and use the distance info to create a centroid.

i found that cvtools does a good job of tracking as well once you fill in the missing alpha of the depth mask. meaning with cv tools you can track two hands its not as smooth as this color tracker. with cvtools you need to blur out your tracking blobs to get rid of jitter.

have you tried cvtools find edge or lines yet. its pretty interesting if you do a peace sign for instance you can place a photo easily on the center or start or end of the line. so not tracking blobs but line centers is also another approach.

I hadn't thought much about multiple blobs of the same color, but yeah, you might be able to adapt this composition to do it if you could partition the image into areas containing different blobs. Or it might be easier to start with one of the OpenCV blob tracking libraries, like cvblob.

@gtoledo3- thanks for the fps numbers. I'm on a MacBook Pro with Nvidia GeForce 8600M.

frame rate is not an issue for me either. here is a game i made for fun using this centroid tracker today. not much concept dribble basketball and throw basket ball but it only took a few hours to set up. i'm happy with the results.

How can i transform another "layer" in the depth map to be transformed into green? I've been trying over and over but i'm not finding any solution.

Thanks!

The kinect patch depth image has been inverted so to get this kernel to work so you can track green for let's say your hand you need to invert the depth image before running through this or alternatively you switch the if statement from <0.1 to if >0.9. if you want to track depth layers and replace with a color just copy the kernel set to .7 or .8 to lets say turn your body green. Its very sensitive even if you can not tell by the depth image. Like .85 will give you elbows if your hand are in front of you etc...

This worked just fine for me when it was first posted, have not touched it since, but I cannot get it to work now. I posted a test movie, and there is a sprite moving in a circle in the comp for testing as well -- two simple tests! Any help is greatly appreciated.

I don't think the composition is correct/finished, after 'Reduce to Single Pixel' doesn't make much sense to me. I am very interested as well how really should work this.

@jersmi - Does the original composition (color region centroid 0_1.qtz) still work for you? The composition you posted is just a little different and it doesn't work for me either. You could try going to File -> Compare Compositions to see what's different, and either change yours until it works or the original until it breaks.

this centroid patch works good for me. here is a macro implementation, you got to remember to use the the transformed image output if you want to do any sort of augmentation as that is the re-sized image used for tracking so if you don't overlay the right image things may seem off ?

the reduce to a single pixel is how this tracks so smooth. from what i understand it is taking a region of interest by using the four corner points and derives the center point of interest and reduces that area down to one center pixel that is then smoothly tracked.

things like open cv etc give are very jittery and usually require a smoothing function to get a good clean track. this patch doesn't require this because of the pixel reduction.

@jstrecker -- thanks for the response. the original comp does not work for me now (downloaded again just now and tried as is). after i set color / threshold for a uniquely colored object i am holding up to my laptop's camera, the tracking sphere moves erratically on a diagonal line along bottom left of viewer to top right.

btw -- MBP, NVidia 9400/9600 GT

one thing different (all i can think of): at some point probably after i updated the Image PixelS plugin to Image PixelS2. However, reverted to previous version does not fix.

What could it be?

Try testing with something other than iSight (?).

See if you make a Render In Image with a a small sprite inside (and clear), and make the sprite the exact same color as the search color, if stuff tracks as expected.

Preprocessing can help, depending on the color of the object. If an object is really reflective, it can seem like there's a great deal of variation in color with lighting and make the color tracker seem not so good. Doing median or mean filtering, gaussian, etc., can help even out the colors of the source.

If it used to work really great, and now it just doesn't, I'm not sure...

Thanks, @dust, for the demo -- good to see that. GT, I set off using a PS3 eye, as you requested.

Then~

Appears that on installing the new Image PixelS2, the plugin reverted to the type "RGB (luminance)" setting for any comps using the plugin on my machine. I changed the three in the comp to R, G and B, and now it has come back to life, though there is an offset in the tracking of the objects, in both the original posted comp and Dust's demo. Still, phew! Driving me crazy, this one! I'm looking in to the offset now. Thanks, folks! (Does it make sense to anyone that this could happen?)

I am experiencing now an offset of the tracking point, which was not present when I first looked at this comp.

Question: what were the original Image PixelS settings? I am assuming R/G/B, but now some adjustment needs to be made and I don't quite understand what needs to happen...

Yes, the original PixelS were (top to bottom) R, G, B.

Thank you. Almost a double post, sorry but I'd really like this issue resolved -- I posted the comp with notes on the offset in the thread by the creator of Image PixelS2: http://kineme.net/forum/Discussion/Programming/ImagePixelS2

Relevant for one because I changed the math expressions in the posted comp to be accurate on my machine (with original Image PixelS, isight and uniquely colored object). Not sure why I needed to change this as I believe it worked accurately before.

Advice welcome on the offset issue for Image PixelS2 as well as your mention of the transformation necessary for different aspect ratios for accurate tracking.

Sorry more questions, trying to understand:

1. might I ask how the math expressions were derived for x,y positions?

2. In the comp you mention that Image Pixel should be equivalent to the Image PixelS plugin. Mr. Oostrik's website says his plugin is based on the optical flow downloader patch: http://www.magdatt.nl/software.html. So I am not sure how the built-in Image Pixel would be expected to achieve this?

This shows how to use area average origin and extent to divvy up screenspace into 4 different regions. When one triggers a color match, one will trigger one of 4 differently colored spheres, and track in that quarter of the image region.