Unsupported — We cannot guarantee that this software will work properly on Mac OS 10.8 and above. Please be careful.

Release: CVTools, v20080207

Release Type: Beta
Version: 20080207
Release Notes

This is the initial beta release of our OpenCV Plugin.

There are some performance improvements over the previous Alpha, as well as some functionality improvements and bug fixes.

It's now possible to control how QC images are flipped when converting to OpenCV.

You can also add additional points to the optical flow patch without destroying its previous input. Currently, I do this with the Structure Maker patch (because it's already written), but if y'all think an OpenCV-Specific patch would be appropriate, I can make that too.

The OpenCV Camera input was disabled (It's impossibly slow on Intel macs, and it hangs on ppc macs). Placing such a patch in your composition will not do anything.

Added cvFindChessboardCorners -- this is a prerequisite for camera undistortion (unavailable currently), orientation (also unavailable), and perhaps POSIT stuff too (unavailable). If you're familiar with the OpenCV API, or know how to turn a collection of Chessboard Corners into a useful structure for other OpenCV functions, please let me know (I'm struggling a bit to figure out how it works.. the cv demos that use these function just produce garbage for me)

PPC may have some colorspace issues (blue/red tints), please let me know and supply examples so I can test better.

This is a 32-bit only plugin for now, so it won't work in 64-bit applications/runtimes.

Both provided demos require the GLTools plugin, and benefit from the YUV Colorspace plugin. The VideoTrack demo also requires the Structure Maker Plugin for live input. These dependencies are just because I was lazy and didn't feel like swapping everything out to QC-built-in stuff.

Face detection kind of works, but it's so slow you have to use swear words to describe it ;) It's also a bit wonky sometimes, even with identical input to the demos... still looking into that.

PreviewAttachmentSize
OpenCV-20080207.zip1.56 MB
VideoTrack.qtz13.57 KB
HoughLines.qtz9.36 KB

cwright's picture
Image ≠ Image

For some reason, it looks like most of the CV Image inputs try to accept QCImage inputs (and then checkerboard the composition, since it's trying to operate on a fundamentally different object) -- not sure why that's happening, but I'm looking into it.

[edit: figured it out: Image ports inherit from virtual ports, and the check I was making checked against inheritance of a class instead of simply "is a" class. Fixed for the next beta. Sorry about that.]

kimba23's picture
great stuff, but need more info

This is really great cwright, I started playing with it, Do you think you could give a little tutorial on how to use the different patches, (maybe comment one of your examples) also more info on how to use face recognition? Is there a wiki I can look at? This is great anyways

thanks

cwright's picture
docs

For now, the best information would probably be from OpenCV itself. Its documentation and sample source are the only useful documents I've found. If you're not a programmer, these may not be very helpful though.

I'll try to whip up some comments for the sample compositions in the next day or two to clarify.

kimba23's picture
docs

Thanks, some comments on your examples would be great!

cwright's picture
Basic comments

Here's the Video Track composition with notes showing the basic functional units.

Here's a Brief document I wrote that describes what most of the useful patches do. It's not verbose, but it might help you get started.

PreviewAttachmentSize
CommentedVideoTrack.qtz15.09 KB
OpenCV Docs.odt3.46 KB

kimba23's picture
great!

Thanks so much.

franz's picture
YUV colorspace ?

"Both provided demos require the GLTools plugin, and benefit from the YUV Colorspace plugin" What do you mean exactly ? there's a YUV to color patch in the graph (that generates a solid color), but no colorspace conversion (where images components would be treated as YUV instead of native RGB.) Working in YUV colorspace would allow for some speed improvements (as there is less data to process in YUV ), although HoughLines and FeaturesTracking (and FaceDetect) can be optimized by feeding Grayscale images (like you did in your example patch).

cwright's picture
requirements

What I meant is that the sample compositions make use of patches found in the GLTools plugin, as well as the YUV Colorspace plugin. For example, the feature tracking one draws points with GL Point, and the point color is determined by a YUV Color patch. Without these plugins, the demos won't work as expected.

You're correct in that skipping conversions between rgb and yuv would be faster; unfortunately, this is one of the design flaws of opencv: no conversion from YUV to Grayscale (this is a trivial conversion, mind you. just extract the Y components). I may get around to adding that later though...

yanomano's picture
crash report

I 've start to play (a little) with open CV ... Seems to be "playable" on PPC now... Here a crash report :

yanomano.

PreviewAttachmentSize
OpenCV_20080207_crashreport.txt39.81 KB

cwright's picture
interesting

Can you explain what you were doing to get that? It looks like a real bug (not the PPC hangs from the video stuff like in the Alphas).

What did you plug into the QCImage To CVImage patch?

yanomano's picture
a video...

a video...quicktime photojpeg 720X576... And i was closing the composition because the viewer was suddently black...

With The houghline composition I have 10 to 15 FPS With the original movie 30FPS With just the movie thru QCimage to open CV >OpenCV to Qc 15FPS

It looks like just the conversion QCimage to open CV >OpenCV to Qc divide the FPS by 1/3 to 1/2...is it correct ? It seems that the rest of the process is very speed...:)

cwright's picture
fps conversion

It doesn't necessarily divide the fps by 2, it depends on how big the image is and how fast your machine is. Mine (A 1.8ghz MB), for example, can do video->QC2CV->CV2QC->Billboard and still get ~100fps (80-120). Perhaps resizing the video to be smaller will help some for slower machines (especially when you do actual CV work, like feature tracking etc).

Does the composition that crashed always crash, or was it just that one time? Can you e-mail it to me so I can investigate?

yanomano's picture
i lose it !

But it was basically the houghlines compose with a movie plugged in the resize patch >QCImage to OpenCV... I was afraid that I think i have deleted the bilboard and was able to see the video...but it is late ...(it's impossible ?) I redo the same composition and it works (now i don't see the video without the billboard...:) yanomano.

cwright's picture
uh oh

I'm not sure if this is the same problem, but I had it do the black screen thing this morning while doing some other testing. It happened when I plugged the video input into the QCImage2CVImage patch (a resize patch was between the two before). It went black/beachballed for a bit (thought it was going to crash), but then it kept going with no image.

After closing it, the iSight light stayed on, and I had to kill VDCAssistant to get the light to go out. While it was on, no apps could use/see the iSight.

Not sure if that's a 10.5.2 feature or what. :)

yanomano's picture
Woo...

I've done some test withThe VideoTrack composition... I have never seen such a precision in a tracker.... The algorythm is awsome... it would be cool to do rotoscoping... or realtime 3D tracking

stefan's picture
incredible work, again ...

actually im working back on my old powerbook and everything is quite slow but i can image how it could rock in future ...

just one question sticking more to the structure maker ... would it be possible to make a dynamicly growing structure maker for the use in iterators for example? that would be really practical to create independent objects ...

best regards from austria, stefan

tobyspark's picture
first proper look: awesome.

first proper look: awesome.

DanieleCiabba's picture
with the opencv can i

with the opencv can i reconize an image from a model?

maybe i can do this with haar detect but i don't know how can i put the image model...

franz's picture
what do you mean exactly

what do you mean exactly ? With Haar, you can detect a face in an image (static or moving). However, CV can't tell Cindy from Claudia, so it won't detect any model....

cwright's picture
with enough training

You can detect different faces if you give CV enough training data to know the difference. I don't know much about facial recognition, so maybe Haar isn't the best algorithm to do this.

I haven't made a way to train CV from QC. You need hundreds of images for it to work, so I figured it would be better for someone to use a separate application for that.

DanieleCiabba's picture
I will try to being more

I will try to being more specific:

I would like to recognize different shapes...

cwright's picture
still training

It's still a matter of training data. The xml file that you feed it allows it to detect a specific "shape" (human face, air plane, box, whatever). You'll need to either change the input xml file to detect each shape (one at a time), or have many haar object detectors (one for each shape) to detect them all simultaneously. Either one will be really slow with the current opencv plugin....

franz's picture
cv request

please add the ability to clear the optical flow patch if possible.

ding's picture
How to download source code

Maybe I'm not looking in the right place but where can I download the source code for this thing. I have done stuff with opencv before and maybe I can help. Thanks.

cwright's picture
source

Typically, betas don't get their source code posted (for a variety of reasons).

If you'd like me to send you the project, we can discuss it via e-mail (I can be reached at the e-mail address at the bottom of each page, info@kineme.net)

franz's picture
filter structure

hi is there a way to filter out the resulting tracked points structure ? i'm getting good results, however, having a structure partially filled with "nan" entries is not so handy... Or is there a "clear" tracked points option in the making ? thxx

edited: sorted, in a way, managed to build a clear track points options myself... thanks anyway. this plugin roxxx

cwright's picture
good feature

I run into this too; I'd like to make a "toss lost points" option, if it's an ok thing to do. I'm not sure if the tracker can re-establish track points when it finds them again (something goes off screen temporarily, or something gets occluded temporarily). Worst case, I can track them internally, and just not export them I suppose.

Thanks for the idea! :) (slowly getting up to speed on cv matrix functions too...)

franz's picture
+ 1 toss lost points function

please please... i'm clearing results with a black swapping image, so it tracks 0 points... (and my structure is filled with nan, but not reduced to zero entry). The tracker does seem to remember tracked points (out of bounds), as even with a "pseudo-cleared" structure, some points DO seem to come back (on PAN shots for instance). Then, after a couple of hours running the patch (heavy use i must confess), my clear trick stops working, for an unknown reason (full buffer ?!).

Anyhow, your GLpoint/line structure renderer is just a GODSEND !

franz's picture
auto-track issue

after some fiddling, it seems that it is the auto-track node that can't toss points. I can clear hand-picked points, but not the auto ones. Any trick ?

If the Clear feature is not an option on the CV side, do you think the GLpoint/Line patch could internally zap the nan. entries ?

cwright's picture
not hard

it's not hard to add, I've just been strapped for time; it's quite simple to add these, just needs a little retooling inside the patch. adding an option to the GLTools patches to ignore nans could be interesting... not sure if that'd make it handy in other places...

franz's picture
worth mentioning

hi, i know you're all uber-busy, but i just found this, openCV like, but more specific functions:
find Heads, find Eyes, find Blinkin' eyes, find heads, even when occluded (whereas CV can't)

here: http://mplab.ucsd.edu/grants/project1/free-software/mptwebsite/screensho...

it might be worth just having a look...

cwright's picture
next round...

The next beta will have at least one cool new feature:

Fast(ish) face detection!

PreviewAttachmentSize
fastFaceDetect.jpg
fastFaceDetect.jpg62.26 KB

mrboni's picture
This is really exciting. Are

This is really exciting. Are the plugins and compositions Leopard compatible? I get lots of errors when trying to open them.

Cheers, Will

cwright's picture
leopard only

These pluigns and compositions are leopard-only. The messages probably mean you installed them to the wrong directory. Please read http://kineme.net/HowToInstallCustomQuartzComposerPatches Before continuing.

[This so needs to be appended to everyone's first post.... sigh ;)]

branchat's picture
Another approach using the OpenCV Framework for Xcode?

Hi,

Amazing job done here! I wanted to connect OpenCV and QC and I found this project by chance. I've tried the feature detector and the hough lines detector... Very nice and works very fluent! :O

The approach I wanted to implement to communicate OpenCV and QC was just create an image translator (like the "QCImage2OpenCVImage" patch), and implement the OpenCV operations in a QCPlugin in Xcode using the OpenCV Framework these guys nicely brought to the mac:

Project page (in german): http://www.ient.rwth-aachen.de/cms/software/opencv/

Direct file download: http://www.ient.rwth-aachen.de/~asbach/OpenCV-Private-Framework-1.1.dmg

This approach will avoid to export the 150+ functions of the OpenCV library by just translating the I/O images. Moreover, it will allow to reuse the existing (written) code instead of cloning its functionality onto the QC graphic language.

Any thoughts?

Thanks,

Rob

cwright's picture
that's what we're doing

That's essentially what we're doing already -- we're not rewriting OpenCV, just using it and making QC wrappers. However, the OpenCV API is terrible, and the library is rather inefficient, so we're looking into alternative CV libraries.

Rony Kubat started working on a QCPlugin-level image translator for OpenCV, not sure how far he got though (he's a user here, and frequents the QCDev mailing list, so you can probably find him somewhat easily to discuss this more)

branchat's picture
QCPlugin-level image translator for OpenCV

It was this "QCPlugin-level image translator for OpenCV" approach I meant. It's not being that easy. :S I'll comment it to Rony Kubat. Thanks! :)

gplloyd's picture
Deleting all track points

Hi there,

This project is great and i'm having great fun trying to implement a sonic mapping for it in SuperCollider. One thing i would really like to be able to do is clear all the points being tracked in "OpenCV Calc Optical Flow Pyr LK". I know there is a "Load trackpoints signal" input but this clears all the points and adds one - i just want to be able to clear everything in the structure. I'm fairly new to QC so there might already be a way of doing this(if so sorry), but if there is please share.

Many thanks,

gpl

cwright's picture
next version

I don't think this version supports that (oversight on my part) -- the next version (which isn't released in beta form yet... (could be soon though, now that I think about it)) supports this feature.

if you'd like to give it a spin, drop me an e-mail, and I'll send it your way. Or, wait a week or so for me to prep it for another beta release.

gplloyd's picture
Cheers

Cheers

kimba23's picture
Re: next version

Any updates on the open cv patch? is there a way to get a new beta? thanks

kristopf's picture
Drop all points / drop lost points

I've been using this patch extensively and I'd like to thank you for writing it.

Also, if anyone would like, I've devised a workaround in javascript for dropping lost points, as well as a workaround to reset the tracking patch, to stop tracking all points.

If anyone is interested I'd be happy to post examples.

franz's picture
+1

I am. I never found the time to find a real workaround for these issues. I'll be gratefull to you if you had the kindness to post such a javascript.

kristopf's picture
Here you go

Hope it's useful.

PreviewAttachmentSize
OpticalFlow_DropPointsExample.qtz19.74 KB

franz's picture
thxx

thank you very much.

gtoledo3's picture
Wow, thanks.

Wow, thanks.

leegrosbauer's picture
composition restoration issue

Hi folks

Can anybody advise on why I'm getting a 'Patch with name "KinemeYUVColorPatch" is missing' non-restoration message for this composition? I have the Kineme YUV Color plugin installed and the Patch Creator does indeed list a patch from Kineme named YUV Color. Do I perhaps need to rename the plugin or something?

Thanks in advance.

cwright's picture
old version

you're using an old version of the plugin (internally, the patch name changed from "YUVColorPatch" to "KinemeYUVColorPatch" to avoid namespace problems with future developers). The new version intercepts the creation of the old version, and automagically creates new ones in its place (so updating is automatic).

This looks like someone saved a composition with the new version, which the old version can't use/understand (which is expected behaviour).

leegrosbauer's picture
probably, but ...

hmmm. I re-downloaded YUV Color but I'm getting the same results. Could you take a sec and check the file in your download link? Either it's still the old one or I have other issues for consideration. Or maybe the new one isn't public beta yet?

Thanks

cwright's picture
seek & find

http://kineme.net/QuartzComposerPatches/YUVColorPatch/0.2

You probably have 0.1 -- up until a few seconds ago, 0.2 didn't show up in the menu (due to a misconfiguration I just noticed...).

try 0.2, and it should work.

leegrosbauer's picture
Got it and

it's working just fine now. Thanks for straightening out the menu. I finally found it just before you fixed it by opening the menu item in a new browser window. That exposed version two.

All the best and thank you.

leegrosbauer's picture
great !

Kris, thanks so much. This is slick!

Salut!

gtoledo3's picture
I just can't help myself...

...

... and a good night to all :o)

PreviewAttachmentSize
gt openCV 3D.zip5.07 MB

gtoledo3's picture
Re: I just can't help myself...

Sweeeet.... because that fella posted, I now see this zip! I lost track of this exact version of the file. The great side effect of posting stuff on Kineme :)

pelican6's picture
Re: Release: OpenCV, v20080207

Does anybody try to use this OpenCV custom patch on Snow Leopard (Developer Preview)? When I open compositions, I got bunch of error messages like these.

(null) : Patch with name "OpenCVBGR2Gray" is missing

Macro Patch Cannot create node of class "OpenCVBGR2Gray" and identifier "(null)"

I've been using other custom patches without any troubles on Snow Leopard.

Any ideas??

mdpotter's picture
Re: Release: OpenCV, v20080207

I'm having the exact same problem. I've got several other custom plugins and patches installed and they work fine but the OpenCV patch doesn't seem to want to function.

I had another problem with TuioClient.plugin too where it would simply quit QC on launch.

I've tried both my home folder library and the root library (not the system library) but neither work.

If there is any more information that you need, please ask as I just purchased the kineme 3D plugin and was hoping to use it with this on an upcoming demo project.

Thanks,

Mp.

PreviewAttachmentSize
Screen shot 2009-11-30 at 3.58.59 PM.PNG
Screen shot 2009-11-30 at 3.58.59 PM.PNG358.2 KB

gtoledo3's picture
Re: Release: OpenCV, v20080207

You're using the old OpenCV plugin that was made for Leopard. That one will not work in SL unless QC is booted in 32-bit mode (do a "get info" on the app icon and click 32 bit more to run older plugins that haven't been compiled for 64 bit).

However, the most current version of that plugin is CVTools, and should work fine in SL, and has some improvements. There may be some patches that have been eliminated from it, but they were largely pointless and shouldn't impact use. Go to the downloads page and download CVTools instead of what you have, long story short.

Installation Instructions

Place the plugin file in
/Users/[you]/Library/Graphics/Quartz Composer Patches/
(Create the folder if it doesn't already exist.)