Augmented Reality in OpenCL (Composition by M.Oostrik)

Author: M.Oostrik
License: Creative Commons Attribution
Date: 2011.07.21
Compatibility: 10.6
Categories:
Required plugins:
(none)

or

OpenCL Combined: Optical Flow, Fluid Simulation & Particle System

I wanted to have a similar flow functionality as Memo's ofxMSAFluid: http://memo.tv/ofxmsafluid.

Looking around a found almost all the functionality on or via kineme.net:

So it made sense to return the result.

Here it is! With a little run through in the attached video.

Note: I created this video using a geforce 480 (with 480 cores). The optical flow and fluid simulation have a resolution of 512x512 and there are 1.280.000 particles. (some visible, some not)

I'd appreciate feedback on optimal settings for your systems. (the attached patch has a resolution of 128 with 128.000 particles)

cheers!

.lov.'s picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Awesome, thanks for sharing!

M.Oostrik's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Boid Like Funny Particles:

dust's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

sweet thanks for returning the shares ties. really nice work. runs pretty good @ 128 res... 20fps

i think this should be able to be optimized by using two video cards. one card for the fluid images and the other for particles and meshes. not sure how else it could be optimized. everything is in cl. maybe making the image resizers in open cl and queuing up your flow images in parallel might improv performance.

either way nice work my friend.

on a different note ties... my friend in japan sent some files to my computer. i told her to put the files in my computers dropbox. she put them in the application drop boxes folder which nuked everything i had in dropbox including some concept proposals of yours from last year. just thought i would let you know. i hope she didn't cause any problems.

M.Oostrik's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

No Worries Dust, I loose stuff all the time, although it's probably somewhere on my systems ;).

What kind of system are you using? I had some similar results on a recent macmini.

cybero's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Impressive and delicate :-)

offonoll's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Thanks for this excellent sample! It is a master composition of OpenCL.

dust's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

thats on a 3 year old mbp with dual 9600M and 9400M cards. interesting enough it doesn't run at all on my air with a 320M card. which does support open cl and ran cl particle patches fine in snow leopard ? kind of bums me out actually, i have already had to send it back to apple once for repair. i picked the 320M as a base model to test CL apps with. that way i could optimize my apps to run on non pro machines.

benoitlahoz's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Thank you so much for sharing !!!

I've been following your blog, and was dreaming of it.

On a 2009 MBP 13" it runs at 20-23fps in 128... It IS fluid.

franz's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

seems cool, however crashes my mac + ati 6750M. It just seems that openCL doesnt like ati. too bad, would have loved to join the party. thanks for sharing

harrisonpault's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

I am getting instability: freezes and crashes with this on a 2010 iMac 27 with i7, SL 10.6.8. Forgot the gpu but it's supposed to be OpenCL capabable I believe. So, what's the protocol? 1) remove all plugins, 2) try in both 32 bit and 64 bit modes, 3) reboot, what else?

Funny thing is, it worked once or twice for a bit. Other symptom I haven't seen before is a frequent blank popup window upon starting the viewer window.

Help?

-Paul

M.Oostrik's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

There are no plugins in the patch (99% sure). So leave them be ;)

I guess you have the ATI Radeon HD 5750. I try to avoid the Radeon cards with openCL because they always give me trouble. (that's why i build a Hackintosh with a Geforce 480 :)

If you make a quick composition with the OpenCL Context Info patch, publish one of the outputs and run the thing you can see if your graphics card supports openCL. (See first picture, if it says CL_DEVICE_CPU like this one your f*****)

This Composition does so much OpenCL that i absolutely discourage everyone to run it from CPU. Even my solid and reliable (but slightly outdated) MacPro freaks out big time.

If your G card does support CL (CL_DEVICE_GPU) try deleting parts of the comp to see where the problem is. Try running only OpticalFlow, then add Fluid Sim and last the Particles.

I would really like to know of there are any bugs so i can improve the patch.

Also what helped for me was to disable Caching for QC: alt + click the quartz composer preferences and go to (Editor) (see second picture)

hope it helps!

M.Oostrik's picture
Re: AR in CL: generating nice textures / shaders?

I would like to work on the visualization of the particles. I have the feeling i should use GLSL for this, but i never worked with this language / QC patch.

Anyone has some pointers for me where to start?

Thanx

mpjstuff's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

@dust, (sorry if this is off topic folks)

If you are actually using the "DropBox" -- then I believe even if something got nuked -- you should be able to login to your DropBox account on the Web and "roll back" the changes. Kind of like a "version history" -- it probably has a limit on how long and how many new uploads you make, but if you get to it quick, you should be able to re-download even overwritten files of the same name.

hope that helps,...

dust's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

cool thanks good to know. seems like a handy app dropbox. i don't collaborate enough to make that much use of it but now that i see you can restore files and roll back, it becomes more useful. sort of like a mini cloud time machine or something. @ties eiPixels folder has been restored.

gabemott's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

This looks phenomenal from the video. And the ball is sparks blue particles... can't wait to see it work on my machine but unfortunately it crashes immediately before QC can even open. Tried disabling cache as per your image posted. Hoping there is a solution.

gabemott's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

This looks phenomenal from the video. And the ball sparks blue particles... can't wait to see it work on my machine but unfortunately it crashes immediately before QC can even open. Tried disabling cache as per your image posted. Hoping there is a solution.

harrisonpault's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

What is your machine and configuration? OS version, CPU, GPU?

mattgolsen's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

I use Dropbox a ton for collaboration with Swiftlikeninja. The revisiting stuff is awesome, plus having our work web accessible is a huge plus. I could definitely build a similar setup myself, but it's hassle and worry free.

harrisonpault's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Okay, I am f*****. QC reports 1 opencl device "CL_DEVICE_CPU".

However, I am upset because the machine has a Radeon 4850 which is supposed to be supported (http://www.apple.com/macosx/specs.html) and runs OS X 10.6.8.

I did a reboot, just to make sure some glitch hadn't disabled the gpu. No joy. Is this something to take to Apple? The quartz-dev list maybe?

gtoledo3's picture
Goosh Re: AR in OpenCL (Composition by M.Oostrik)

I started digging into this a little more, because I was interested in seeing how the particle engine part of it would work with the fluid sim that I use.

One thing that I noticed is that it seems as though a great deal of what is happening in the particle system velocity ends up being dictated by the initial optical flow. At first, when I started looking at the particle engine part, I couldn't figure out why I wasn't getting results from using the flow from my own liquid sim.

I've added two parameters to your composition, to influence the weighting of the optical flow image, vs. the fluid image in dictating the position of particles. Ultimately, what's coming out of the liquid sim itself is a bunch of vectors that dictate the flow, so it can be constructive to amp that up, or diminish it, and let the initial force of the optical flow prior to the liquid sim have more influence. I have left the liquid sim part as is.

With this, the particles are more gooshy and move in a more liquid like manner.

gtoledo3's picture
Re: AR in CL: generating nice textures / shaders?

M.Oostrik wrote:
I would like to work on the visualization of the particles. I have the feeling i should use GLSL for this, but i never worked with this language / QC patch.

Anyone has some pointers for me where to start?

Thanx

What were you thinking about... do you mean for texturing of the particles, or something different?

M.Oostrik's picture
Re: Goosh Re: AR in OpenCL (Composition by M.Oostrik)

I can't watch your version right now because i'm in the process of updating my machine to Lion (kind of windows XP like process with a HT ;)

I like the idea of having a couple of parameters that result in different particle behaviors. It's more or less what i did with the magic particles video. I took it out of the comp to keep it as straight foreword as possible.

It's indeed visually rewarding to play with these particle forces / velocities.

Cheers!

M.Oostrik's picture
Re: AR in CL: generating nice textures / shaders?

I'm thinking about a sort of glassy feel to the particles, with a simple light setup and maybe some refraction.

M.Oostrik's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

We were talking the other day about this: http://kineme.net/composition/dust/parallelpainting

It's a shame ATI and OpenCL don't work well together

I still have my hopes up for Lion, but this could be just wishful thinking.

harrisonpault's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

I posted my concerns to the QC list, and also did a search on said list. According to an earlier post by C.Wright QC requires some additional features beyond OpenCL standard support and only certain (unnamed) cards support the (unspecified) required extensions. Other's have suggested that these mystery requirements are "image support" or "texture buffer" support. But neither of these terms clearly map to extensions named in the OpenCL 1.0 and 1.1 Specifications, as far as I can see.

In any case, C.Wright makes it clear that other, non-QC, applications may use an OpenCL supporting gpu, even if QC's unspecified secret additional requirements are unmet by the card.

Thank you, Apple.

Anyone (inside Apple or outside) who can shed specific light on this? Answers could help customers make informed choices on hardware purchases.

gabemott's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

I'm in the same boat: "F*****" (CL_DEVICE_CPU)

Real bummer as this is on the newest MacBook Pro 10.6.8 AMD Radeon HD 6750M Intel HD Graphics 3000

Hoping Lion might improve but unlikely eh? Are there any possible solutions for this machine?

dust's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

well i got CL working on my 320M by restoring from lion and not installing on-top of a fresh snow leopard build. unfortunately now camera input doesn't work so i can not run this patch. if its not one thing its another thing. my camera settings are greyed out. not able to even use an external cam to test with. i had about 5 min of WOOT WOOT as procedurally cl is working fine but now its BOO BOO defective malfunction. honestly can not tell if its hardware or software at this point. ;(

PreviewAttachmentSize
malFucktion.png
malFucktion.png37.72 KB
cam.png
cam.png955.47 KB

dust's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

so i'm testing this with a few systems at the lab...

first test is on mac pro with ATI Radeon HD 2600. the card isn't supported however the cpu render of this patch on a dual quad core intel xeon with 10gb of ram. @128 resolution i'm getting 18-20fps with everything turned on. @768 resolution with everything turned on i'm getting 3 to 5 fps.

second test is on iMac with AMD Radeon HD 6970M with CPU of 3.4ghz iCore7 with 8gb of ram. will not render at all. no mouse no video. simply crash o' la.

will test some more systems later got to run.

usefuldesign.au's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Such an awesome looking comp. I can't run on my old Mac nor if I by the newest Macs it seems :-\

I'd really appreciate it if somebody could build this composition into an App using Quartz Builder and post it. I'll take it into the Apple Store and try it on every machine they can throw at it. Mini to Mac Pro. I'm on Leopard so can't do myself.

It' so frustrating to be uneasy about spending on the latest hardware (was thinking latest MBP 17" or latest 27" iMac). If it weren't for Sandy Bridge CPUs and TB i'd be down for an Nvidia 512MB GPU MBP reconditioned/2nd hand for sure.

usefuldesign.au's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

So I just took the .qtf file into the local reseller and open it in Quicktiem Player app. Crashed QT app on every Mac there. Nothing with a NVidia GPU.

M.Oostrik's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Yes that is a shame...

The last affordable mac that will play this comp is the last gen MacMini (mid 2010) It has a Nvidia 320m.

I have one here and it performs well on low settings, 25fps at a resolution of 128.

dust's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

here is app i built in lion 64bit with apple llvm compiler. tried making a quartz builder app but it doesn't seem to be working in lion with this patch. running a qtz file in qt doesn't work anymore unless you have quick time 7 installed.

PreviewAttachmentSize
Audmented Fluid Reality.app_.zip179.76 KB
AugmentedFluidRealitySRC.zip134.23 KB

usefuldesign.au's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Thanks Dust, Note that I did take in a few simple demo .qtz compositions of my own and they all ran okay in QT Player app (presumably Lion it was an Apple reseller so I assumed it was Lion).

You get a dialogue saying the program that made the file is not installed but you can run in Quicktime Player if you want to. Like I said some simple .qtz files worked fine not sure which version of QT the player was using.

cybero's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Thanks for the example dust, I tried creating a QB application using m.oostrik's original, but found it only partially functional.

Thankfully I find other compositions don't fare quite so badly in QB, even if depending upon one OpenCL kernel or another, bit of a mystery as to why a seemingly supportable input to QB should produce such a partial output.

Thanks also for the skeletal project.

M.Oostrik's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Thanx for the app Dust!

Tried to find out how it works , but i don't get the new Xcode. Can't seem the find any code that links to the quartz composition. Where is the comp anyways?

Can we do a Skype Session?

Cheers M.

rui.monteiro's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

very nice work!!! ;)

dust's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

I instantiated the comp into the xib. so there is no need to load it from a file just needed to add the qc framework. You can do a lot of things with story boarding now. Story boarding being formally interface builder. you can do thing like dragging an object to your app controller and having the property methods generated for you etc... yeah I'm down for a Skype anytime.

gabemott's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

Just coming back to this to see if there is any chance some miracles have developed so this would run with the graphics cards in the newer macs. Looks unlikely. I'm also wondering if there are other compositions out there that are at all similar to this one...

jean_pierre's picture
Re: Augmented Reality in OpenCL (Composition by M.Oostrik)

i hear you, it is really quite unfortunate and un-amusingly ironic that the entire current line of Macs, ship with AMD graphics and in turn cannot run the composition's OpenCL patch. at one time i thought QC was the tool to develop OpenCL in, but given the woes with AMD graphics, there must be some other environment this is done in?

initi's picture
Re: Goosh Re: AR in OpenCL (Composition by M.Oostrik)

it's crashing down on MacPro, ATI Radeon 5770. don't anybody knows why?? Thx!

cybero's picture
Re: Goosh Re: AR in OpenCL (Composition by M.Oostrik)

On 10.8, suspect the component part, Density OF Mask patch set, as the Optical Flow worked in use inside the Particles Boids composition.

I'd also suspect the Occluder and Pressure OpenCL patches.

They also failed in being reusued elsewhere.

Haven't checked them in painful isolation as yet but that's what I think would help figure what wasn't working even when not in any odd mix up in QC.

Probably down to Graphics Driver updates since the composition first got published.

PostScript

Just checked these OpenCL kernels in isolation and found that they all work well in isolation, all excepting the 2d Fluid Simulation macro patch, then again, that might well be due to the resolution not being transferred correctly - like maybe its sampling 256 but set to 128 Resolution. Just a thought. The correct resolution did sort that problem out.

Current conclusion, the OpenCL Macros don't play well together, suspect some mismatch of resolution or image type, unscaled, unsized or something. Given that trying to work the original or forked AR .qtzs, I'll approach this as a rebuild, adding them back together, step by step.

PostPostScript

Just rebuilt the item and it breaks down at the Particles patch.

Still to employ the BrushImage patch.

Everything else works. See attached example.

Might try replacing the Particles patch with another stable on its own Particles patch and see if that works nicely.

PreviewAttachmentSize
augrealrebuild.png
augrealrebuild.png850.72 KB
camera4.qtz161.93 KB

initi's picture
Re: Goosh Re: AR in OpenCL (Composition by M.Oostrik)

ooops, I missed this post, but thank youuu!! did anybody make some progress on this already? it's completely freezing whole system (mountain lion) if i connect any "rendering macro" from original M.Oostriks patch (particles, velocity flow, velocity fluid) I'm running MacBook Pro retina, 10.8.5. thanks for any help in advance!