AudioVideo Patch, v0.2

cwright's picture

The AudioVideo 0.2 patch has been released. This recently-discovered crash problem has been resolved, and some performance improvements have been employed.

toneburst's picture
Feature Request: Channel Select

Hiya,

I have a feature-request: I'd like to be able to choose a specific input of a multi-channel audio input (for example the Soundflower16ch virtual input).

alx

Quartz Composer Blog: http://machinesdontcare.wordpress.com

Music Site: http://www.toneburst.net

Roger's picture
No output at all?

First, two thumbs up for the effort of making QC support multiple cameras of similar brand! Being able to have parts of our stereo vision project in QC would be so nice.

Unfortunately I don't get any output from the patch. :( Even when I set the video device to the single iSight and feed the output image to a billboard, the only thing showing is the checkerboard background. Using a sprite, instead of the billboard, results in the sprite showing but with uniform color and no video. I also have the occational QC crash but that is only expected from v0.2 software.

I'm running QC 3.0 under OS X 10.5.6 using a MacBook Pro (not unibody version) with 2.4 GHz Core 2 Duo and 2GB of RAM.

Any hints?

/Roger

cwright's picture
don't use?

hint = don't use this patch? The problem it tried to work around is actually a bug in QuickTime, so it really doesn't serve any purpose. Use the built-in video input patch.

Roger's picture
Want to use :-)

But I do want to use this patch since a prerequisite for my stereo vision project in QC would be to simultaneously access two webcams that have the same name, which as you say isn't supported by the built-in video input patch. I'm sorry if I was being unclear about how I intend to use your patch or if I have misunderstood its purpose.

Still, I get no image on the output of a single Kineme video input patch but the cam led is lit indicating its active; it doesn't matter which video device I select. Using two patches (which I eventually will have to do) lights the led on one, keeps the led of the second dark, and gives no image from either.

Do you have any suggestions on how I should continue troubleshooting this?

I have ruled out the webcams as the source of error given that:

  • each webcam does provides an image when individually accessed using the built-in video input patch,
  • accessing the built-in iSight from the Kineme patch gives the same result: a lit cam led but no image.

Thanks for your patience and assistance.

cwright's picture
misfeature

The original intent was, in fact, as you figure: To allow inputs from multiple cameras with the same name, at the same time.

HOWEVER: It Doesn't Work. The reason it doesn't work is because QuickTime is a complete and utter joke. There is Nothing we can do to make it work with multiple cameras, so regardless of whether or not we get this "working", it will Never work for what you're wanting anyway.

I'm sure some recent QuickTime update broke this patch, but since it won't add anything over the built-in video input patch, it's not worth our time to correct it (until the underlying QuickTime bugs that prevent multiple identical cameras from working simultaneously get fixed, which is likely never since everyone's falling all over themselves at how great QT X will be...)

stuart's picture
Re: misfeature Got 2 Cameras working

Hello, I am new to the forum and have already benefitted from the knowledge of it's users, so first and foremost, thanks to all that have shared their experiences and techniques on this forum.

Second, big props to Kineme for their great patches/plugins. I've already bought them all and am very happy with them all. Great job.

I just figured out how to get two firewire cameras of same type to work in QC in 10.5. All it took was installing a PCIe firewire card to give the system a new bus to get past this bug.

I bought the $70 Unibrain FireBoard800-e™ V.2 1394b PCI-Express adapter for my Intel Tower, popped it in, and was pleasantly surprised I could get both cameras to work by having one on the built in bus and one on the card.

I too want to use duplicate cameras for stereo projects among others and was frustrated that something that worked in QC 2 and 10.4, didn't work in Leopard.

I suspect for laptop users, they can use a PCI express card to overcome this bug in QC.

I also bought a FireCard800-e™ 1394b ExpressCard/34 for my laptop, but haven't got it working yet because it turns out I need to get a power supply for it to give bus power to drive the camera. I suspect it will work and will post when I get a chance to pickup a PS and verify it.

I believe it's a bug in QC3 vs. in Quicktime because I could get both cameras to work in Leopard within the Max/MSP environment using the built in FW800 ports. I'm only having the problem in QC3.

Although it's something that should work without any additional hardware, all it takes is a $70 card to get past this roadblock.

Hope this helps others sharing the same frustration

Cheers

Stuart

cwright's picture
Re: misfeature Got 2 Cameras working (facets)

There are a few things with this:

The built in patch cannot work because of a quirk in AppKit: can't have duplicate entries, so 2 cameras won't get added. The AudioVideo patch addresses this by adding the number to each entry (so they're all unique).

Glad the multi-bus thing works. What camera model do you use? In speaking with some QT guys, it seems to also depend on lower level stuff too. I'm not sure if it's in every case, or per-camera/driver.

If you have a moment, please indicate whether this patch helps with that or not (or if it even works at all... I've personally not tried it in 6-9 months, as I have no need for it, so it may have gone stale).

stuart's picture
Re: misfeature Got 2 Cameras working (facets)

I'm using 2 of the Point Grey Flea2 cameras. FW800 1280x960. One color and one monochrome.

I just tried it with the patch and without. It works with either built in or patch as long as I'm using the new bus for one of them.

I tried hooking up a third camera, this one is an Imaging Source FW400 monochrome and although i could now see it with the patch, I could not make it work in tandem with the others.

It seems that all these machine vision cameras have the name "camera", no brand or model in the name.

If there was a low level utility that change the name in the quicktime system based on serial # or some other unique #, that would sort it out.

I suspect if I added a PCI FW400 card, might make it work since it adds another level of hierarchy to the system and gets past this name bug.

The patch does indeed add another counting system onto the name, but it does not fix this problem. Although it might help if a third camera is added via another new FW bus created by a PCI card or similar method. Although I'm not using it now, I'm going to keep it handy in case....

In the meantime, I got two cameras working and am going to continue on with my project.

I'm using the monochrome as a IR only cam to create a live mask and the color as the fill. The idea is a poor man's depth keyer for use in the live touring business. The artist will be in front of an LED screen which I've already tested for IR and it doesn't emit very much at all. I should be able to mask that out be left with just the subject as long as I get enough light on her without getting it on the background.

Cheers

stuart's picture
Re: misfeature Got 2 Cameras working (facets)

Oh yeah, as long as we're talking about patches. I miss the built in Video In patch from QC2 and the extra parameters that it had. Being able to resize the image right at the source was my favorite feature.

Not sure why that changed that. Now we are forced to use a Resize object to shrink our image size, but it still means the capture device is grabbing at native resolution I believe.

It would be awesome to have an enhanced Video In patch that had that as well as handles to the image controls so they could be controlled within quartz.

With these machine vision cameras, alot of tweaking is often needed to get the right results. ie color balance, white balance, ect.

cheers

franz's picture
Re: misfeature Got 2 Cameras working (facets)

hi stuart, could you expand a bit on the Point Grey Firewire Flea2 cameras ?

Are they good ? Are they fully Quartz Composer compatible ? Could you do a snapshot of the pref. pane if you don't mind ?

Also, can you, under Quartz, completly override controls, and be 100 % manual, with option for iris, shutter ... etc. Would you say that it is better than the Fire-I unibrain Pro camera ?

Thanks in advance for sharing your experience.

stuart's picture
Re: misfeature Got 2 Cameras working (facets)

I like the Flea2 cameras. I bought the FL2G-13S2C and the FL2G-13S2M because they use the Sony ICX445 CCD, which is more sensitive for Near IR than the other image sensors. Also they are FW800 and allowed me to get 1280x960 @ 30fps, although the color specs at 27fps at that resolution, 1280x720 @30fps. They cost $800 each.

Unfortunately in QC3, Apple removed the "Show Device Settings" button from the Video Input patch in the inspector pane and we are left with just "Deinterlace Fields, Enable Clean Aperture, and Disable Color Correction" options.

I can adjust the advanced settings in QC2 or Max/MSP, just not in QC3.

I sent some screenshots, the first is from QC3 and the other four are from QC2.

With QC2 I can put all the parameters in manual mode via the Device Settings, but in QC3, I get the dumbed down panel.

I don't have first hand experience with the Unibrain line of cameras although I just bought 2 of the Fire i board remote CCD cameras and haven't had a chance to use them yet. I bought these because they are cheap and the CCD is mounted on a separate board to allow some different mounting options.

Before I bought the Flea2's, I did spend a few nights comparing the Unibrain, Imaging Source, and Point Grey products and found the Point Greys to have the best features for the price. I found the Unibrain's to be more expensive in the 1394b range.

I must say though, I was impressed with the sales guy I spoke with at Unibrain and his knowledge of their product line.

So to answer your question, I can access the advanced camera settings and do manual control, just not in QC3.

I wish Apple would put the "Show Device Settings" button back in the Video In patch.

Hope that helps

S.

PreviewAttachmentSize
QC3 Video In Pane.png
QC3 Video In Pane.png16.83 KB
QC2 Video In Pane.png
QC2 Video In Pane.png40.43 KB
Picture 1.png
Picture 1.png16.13 KB
Picture 2.png
Picture 2.png37.07 KB
Picture 3.png
Picture 3.png18.5 KB

franz's picture
Re: misfeature Got 2 Cameras working (facets)

Thank you very much for this info. It is good to know that settings aren't exposed in QC before buying a 800$ camera...

Chris : do you know why these settings aren't exposed anymore, and if it is possible - and worth - to have them back in the KnM plug ?

Thanks all for your kind patience.

monobrau's picture
Re: misfeature Got 2 Cameras working (facets) +1

Would really love to have this feature back again. I could access my DFG/1394-1e settings in QC 2, but not anymore in QC3. It would safe a lot of adjustments (and some processing power) via QC patches.

mfreakz's picture
Video inputs, FW/USB Cams...

Really, really thanx for all those information ! I'm working on a similar project (using two cameras to deduce distance between objects). Many of my future projects may use at least two cameras, and i'm stuck with this stupid Quicktime matter ! I borrow two iSight for experimenting with two camera inputs. First there is the matter of the "same name", but also (Chris tells me ;) Quicktime couldn't manage two firewire camera (a kind of limitation). You find a good point using a second firewire card in the same computer ! That really a good point for us ! I was think about using a firewire camera and an USB one together. But there is another matter: USB webcam (the picture quality isn't my first prerogative for begining testing) aren't always OSX friendly ! I learn a bit on the UVC standard, but it seems very very difficult to choose a good USB cam for Mac nowadays !

I wonder if there is the same Quicktime's matter with two USB cam ? Do we use two different USB card in the same computer to manage with two USB cam ? If not, could it be possible to use both an USB and a FireWire Cam as two source inputs in QC ? Do you find also a good (and cheapest) USB cam (OSX friendly) ? I think we need a special thread in the Wiki (with list and comparisons). We need to keep this thread alive !

stuart's picture
Re: Video inputs, FW/USB Cams...

In my opinion, if you are going to use two cameras to do depth analysis, you would be best served with 2 identical cameras if you can.

Although the hybrid USB/FW might allow 2 cameras, you will probably suffer the same frustration I had when it came to matching up the optics, image levels, and of course latency of the two different cameras.

It might be worth trying out using a USB hub to trick QC3 into thinking it has a new bus. I've noticed that when adding a hub to a Keyspan serial adaptor, it adds another hierarchy to the naming system. Try one cam directly into the computer and one plugged into the hub.

I thought that might work with the FW800, so I bought a Unibrain 1x3 Repeater, but that didn't sort it out. I don't know if firewire does the renaming thing internally if being used with a hub.

I was determined to get both cameras working in QC. I knew Quicktime would allow it because I could use both cameras in Max/MSP/Jitter so I took a chance and bought a couple different pieces of FW800 hardware to experiment with.

I haven't been able to get 3 different cameras with the same original name yet. Even though my third camera is FW400, it still doesn't access because it's name of course is "camera".

It might be worth talking to the manufacture and see if it's possible to change the name of the camera before they ship it. This applies to the machine vision cameras, if they named it cam_Serial#, that would sort us out.

good luck with they stereo project, it's really nice that computers are finally fast enough to do this stuff in realtime.

cheers

S.

dust's picture
Re: Video inputs, FW/USB Cams...

im not actually sure how you are trying to implement, a 2 cam set up for depth but you could do a simple depth calculation with the open cv beta using on cam. depending on the legitimacy or accuracy needed for the project you could simply calculate the difference between two tracking points lets say the eyes and use that for say a z depth. realistically you would want to do some fancy trig stuff creating a right angled triangle based on the camera fov and subtracting screen difference in relation to the position of the camera or what ever to get more accurate results. i know i have been messing with opencv using the recognition bounds as a simple point of reference for z depth, or in other words just using the area of a square marque based on recognition, not darpa grade but good enough for me to play video games without touching anything.

stuart's picture
Re: Video inputs, Depth cameras

Hey Dust Thanks for the info.

I'm not really trying to depth calculation, but a way to mask a performer from the background in realtime and then use that video for special effects in a live concert environment.

Unfortunately I did some more test the other night and I'm getting too much stray IR bouncing off any reflective surface on the stage to make it work reliably. Also the LED screen behind the artist is indeed splitting out IR which shows up if the image is very bright.

What I was trying to do is kinda like the Zcam from 3dvSystems http://www.3dvsystems.com/technology/product.html

It's basically a camera that creates a realtime alpha channel that can be adjusted for depth which outputs that along with the color camera image, thus essentially removing the background in realtime.

This company first made a $250,000 version of this and has built it into a smaller product which they call a web cam. Although they have been advertising it for years, I have yet to ever see it for sale. I wonder if they will ever release it or are just planning on selling the technology to a larger company to increase their profits from this R+D.

I believe the depth part is based on a Time Of Flight of IR light, which is emitted from the camera itself.

Anyways, like I said, I realized that this is not going to work on this particular tour and with this LED screen unless I put IR blocking film on the LED tiles. Unfortunately our LED screen is huge, over 60 feet wide and it's too late in the game to make this gag happen for this tour.

I have made it work if the performer is standing in front of a projected image, but that's not this tour.

Cheers

PS, I love this forum.

volkerk's picture
Re: Video inputs, Depth cameras

this thread is old now, but someone might be interested to resurrect it.

I tried two cameras on my macbook pro, the built in iSight and a USB Logitech Vision Pro (which is actually a great camera) and I only get either one, both simultaneous does not work. Not with the built in Video patch, nor the kineme video input (which actually crashes if i try two instances)

unfortunately i have only one logitech cam to try it with them.

psonice's picture
Re: Video inputs, Depth cameras

If it's any help, it's possible to set up multiple cameras in a separate application, then pass the stream to QC.

I've not set it up with multiple cameras at once, but I can't see any issue there - so it'd be possible to create a QCPlugin to do this fairly easily if you can do the obj-c stuff.

djoannid's picture
Re: misfeature Got 2 Cameras working (facets)

Stuart, i have the same problem with Grasshoper. Did you find any fix to this?

An other issue is that you mentioned the near-IR of Flea2. Have you tried this sensor in night and what was its performance ? We are looking into purchasing the Chameleon camera (uses the same Sony CCD as Flea2). Do you have any images to send me ? (djoannidis@hotmail.com).

Do you own the color camera and can switch from near-IR performance to normal mode (using the SDK?)

sbn..'s picture
Z-Cam

I just saw this thread, and I, too, have been drooling over the Z-Cam.

The Wikipedia entry for the company that made it says that it's been bought by Microsoft for the Natal project. That's good news in that it will probably be available as a mass-produced gaming peripheral targeted at consumers. It's bad news in that we'll have to wait for mr. Molyneux to get his act together.

ETA: Judging by the Wiimote, X360 controller, and PS3-cam, hackers will love to get this thing working for non-Xbox devices.