Mouse Sensitive Panorama Backgrounds

TheRandomDude's picture

Hello Boys and Girls,

Heres a quick composition for a trackpad sensitive background environment. Plop Pano.qtz into your "~/Library/Screen Savers" folder, then run "Panoback.command".

Things to know:

  1. Replace the test image with anything you like, I found a lot of good images here: http://www.flickr.com/search/?q=Equirectangulars

  2. The mouse has to be manually entered as a UID device for it to pan correctly and the Screen Saver engine not to ignore it. I've set it up for a Macbook trackpad for the time being, but I'd definitely be open to other options.

  3. Please improve and resubmit, this is rather drafty.

PreviewAttachmentSize
Pano.zip3.59 MB

gtoledo3's picture
Some questions... I guess

Some questions...

I guess the shell command is what is enabling your splash? B/c it is looking for a file that isn't "there"... and causing QC to crap out if I render it with that macro enabled.

{ok, how many of you are actually doing the suggestion, and how many are just opening it up in QC? ;O) }

Or is this command line thing simply for your mouse when in screensaver mode? I know this is horrible of me, but I will never ever install this command line, because it is antithetical to my sense of how I think things like this should work... so my apologies if that makes it do some amazing stuff that I am missing out on.

I'm scratching my head here... the example that Chris posted up a month or so ago could be easily setup to be "grabbed" by the mouse. I do like your scheme for movement though... I just wish the "throw" was greater, and that you could still "grab" the image. Maybe that's because it's what I am used to. Your setup is closer to Apple's via Quicktime version I think. As long as you are on that, you might want to check out the PDF zooming example in the developer examples?

I am on one hand in awe of your mouse patch.... yet I don't like it at the same time because it screws with how I set up Spaces and "hot corners", just to get it to rotate all around. I guess I prefer the simple "grab" and move (with maybe some interpolation for a coasting rotation).

What is the plusses of your mouse movement scenario in your mind, I think I might be overlooking something here...

Despite any quibbles...

THIS ROCKS. Great picture selection, inventive mouse patch to say the least...

I'm going to dig some out to post.

TheRandomDude's picture
A Few questions answered...

It sounds like your having issues increasing the movement speed, but remember this is supposed to be passive and keep distraction at a minimum. It's easy enough to change on your own though. The way to use it as a background is with the command, so let me explain that whole deal.

Once you place the Pano.qtz file into the Screen Savers folder, the ScreenSaverEngine application recognizes it as a saver module, and the command line can work it's magic.

It will play it as a moving desktop background that you can passively play while you work on other things. I kept the CPU usage down to about 5%, so it shouldn't interfere with most applications.

Because the ScreenSaverEngine doesn't like mouse input (probably because the mouse isn't supposed to have anything to do with screensavers) using a trackball patch isn't supported. Heres a version that speeds up the environment if you start dragging, but remember that dragging works in any application no matter what your clicking on.

PreviewAttachmentSize
Pano2.zip9.76 MB

gtoledo3's picture
Oh I see... I just keep a

Oh I see... I just keep a script on my dock to do the whole desktop deal. Now I am getting it! Totally understood why you would want this kind of throw to it. Great call!

I forgot about the part of the screensaver thing, because I tend to not use the screensaver function.

Clever as heck. Double kudos!

TheRandomDude's picture
thanks :D It was a lot of

thanks :D It was a lot of workarounds (my favorite kind of work).

dust's picture
panoFlow

i tried the the emended version of the the pano patch and maybe your not supposed to open in quartz but it wigs out on my system. haven't tried running the mouse command. don't know why the world is turning all upside down and stuff.

here is my version. i have been playing with panoramas lately. great flikr link by the way. i had been making my own panoramas in 3d but photos look pretty good.

try this one in as your screen saver you don't even need the mouse. move left, move right move up and down in the room and the screen saver panorama will go in the direction you are going in. it works best if you actually stand up and move back from your computer. maybe at a 5 + feet from the screen.

this will only work supposing you have a camera. it uses the optical flow example included in the developer examples/plugins so it may work without the plugin but you can just open the example and build the plugin. and of course it is leopard only.

thought if you take take out the mouse equation 1, you won't have any of those hot corner expose space issues. give it whirl you should be able to pan 360 on x and only 180 on y. waiting for the next release of openCV maybe it will follow your head or something instead of just directional velocities.

if it spins 2 fast and you get sick then adjust the flow steps and iterations to suite your environment. the image must be pretty large 9.8 mb

PreviewAttachmentSize
panoFlow.zip9.76 MB

leegrosbauer's picture
nice

Thanks folks! and .. panoFlow is fun when bounced, even. :-)

TheRandomDude's picture
Yeah, this is pretty

Yeah, this is pretty impressive stuff but as an interactive background it definitely sucks up a lot of CPU. I've noticed you have to cut down the multiplication of relative movement by a lot if you want to use a mouse, compared to that of a trackpad. It may be the case that the powerbooks need some adjusting as well?

I do like the idea of using a camera, just image processing can be such a horror show for the macbooks. What I'd like to see more of is low "cost" quartz compositions that can work as interactive wallpaper. The idea of using a camera is an excellent idea, but is there a way to make it take less than 10% CPU?

gtoledo3's picture
You can try putting in a

You can try putting in a render in image and "downsampling" the image size... But it may look like garbage, especially since you are talking about running fullscreen.

I think if you just use the sphere and attach some interpolation/lfo for the movements and forget about all of the mouse stuff, you would get a nice fps. Try using the kineme fov in conjunction and see if you like how that plays with this kind of setup.

TheRandomDude's picture
Alright

But really, whats the fun in that? Although these spaces are beautiful on their own, I think it's more interesting to integrate interactivity into both an active and a passive role. Thats the real trick with something like an interactive background.

gtoledo3's picture
Ok.... after Chris posted

Ok.... after Chris posted that panorama example a couple of months ago, I have been mucking with panorama's. Some of my web clips have used the panorama background, because it is just a really nice way of establishing a realistic looking background for something to be animated on top of, that is easy on the computer.

That being said... I believe that people are so wowed by the panorama's that someone could probably make a living going around the world, taking panoramic pictures and selling them on a computer program (and it would probably be better to do it in something besides QC frankly, for cross compatibility).

That being said, this being said, ( :op ok, I am playin' with words sorry) ....

I would set up a website with an RSS of picture url's of whatever I liked, so it would be a "changing" program. Or my own website with whatever pics I wanted ( I would possible do it that way if I was really "into it").

So.... the RSS would be feeding url's to a "web image downloader" type of plug in of some sort. You would feed the image through a chain to always make it a standard size before it hits the sphere.

Then you would publish stuff like the "refresh rate", which would be controlled by the speed of your rss, as well as the internal refresh of your web image plug in (probably shouldn't publish that, and it should be set slow for CPU's sake, since most people probably don't need to have images changing every second or anything like that)...

THEN.... I would setup 3 (or more) control modes. Something like the "autopilot" mode that I describe, something with a trackball type of idea, and then something like the optical flow idea. Those choices would be published. I would also publish speed and rotation controls for the "autopilot" mode, so that the user could change the settings in their screensaver preferences.

And of course, last but not least.... I would just have to plop a giant teapot in the middle of the scene ( ...joking.....maybe).

gtoledo3's picture
mmm... tea

I've done ton of "render" using panorama, but here are a few that I dig the most...

When Kineme3D is full release, I don't have a problem posting a couple actual files, but I don't want to do that without the the ok...

.... this one uses Chris's Kineme3D battle comp as a macro, with some of Steve's particle macros, tweaked a little bit. This one is a actually a pretty good example of the new build of Quartz Crystal's motion blur as well.

A couple of my first experiments from a few months back... these used the teapot, partially to annoy Chris, and partially as a way to see if I would get any weird z fighting problems if I tried to plop actual 3D objects in a panorama setting.

EDIT... well, one of my first... I guess myspace vids can't be embedded...

dust's picture
panoPot

tea time anyone. im a coffee man unless im abroad. but george thanks for bringing kineme 3d to my attention. bought it yesterday. going to post some screen stuff in a few. pretty sick fluid dynamic simulations with real time play back via quartz. is what im bout to try and accomplish, just got to figure if the .fxb export will do what i want or if i got to use the .md2 format in the examples. nice texture on the teapot george, im excited bout the kineme 3d plug. looks like your getting the pano reflections and everything. i suppose i shouldn't have spent a whole night simulating particles and waiting for them to calm down just to test the .fxb but figured if im going to test 3d, and converting .sd 2 .fxb might as well use something other than a teapot. lol. defiantly want to look at some of your patches george when 3d final is released. strange the .obj is slow i tried the obj loader and seemed be fine for me, going to take chris's advice and use .fxb its funny lizard man farts on the knight. lizard farts are no joke, i got two 5 foot iguanas, or well my wife does in my bedroom. i let em loose last year and un domesticated them, sort of they still use the potty but being locked up is no fun. just had kid now there on lock down again. i think i would be more of the ghecko type or dragon type of lizard man if i had to choose.

dust's picture
rss pano feeds

i actually thought about doing that sort of thing. or sort of. putting some panorams in a director and randomly changing the image of the panorama at x time or what ever. actually yesterday was the first time i put a qtz patch into the screen saver. even doing some sort of transition effect between panoramas or what ever. maybe you can all ready do that like the desktop images or what ever with multi qtz files i don't know have to mess with that later. until now i have only tried the visualizer stuff but it seems like the screen saver will take a cam input so maybe it would take audio as well. could be a cool collaborative program where people download some sort of optical flow panoramic 3d aduio visual screen saver or what ever that randomly changes pano and will react to the rooms noises as well. not sure about making a living on it, unless it could be ported to the iphone, at least i know thats mostly where i buy random apps that i don't need. but then agin a screen saver would suck battery, so it maybe be better for a website, will have to investigate optical flow integration into flash cause i know quartz can't get audio or camera input from web. is it just me or did action script 3.0 take an entirely new direction ? i still use 2.0 seems silly to me to write a function to call a button when before you could just click on the button and insert go2frameX. thats the life got to keep up with the technology.

cwright's picture
obj vs fbx

If obj reading is fast enough for you, then feel free to continue to use it. It's only on really large models where it starts stalling things for a very long time.

dust's picture
fxb.obj

well i tried the obj loader plugin, not sure if you made that or not. didn't try and iterate through frames of animations with textures or anything just loaded in a terrain. i can imagine why obj is slow, it can include everything from textures 2 dynamic info, skeletons and stuff, would be a head ache to sort all that out in a qtz patch. i think im going to shoot for a .md2 or .fxb, bout 2 hours away from saving my initial state and rendering my particles to mesh. if autodesk can sort out the obj issue im game but im happy just to be able to get models into qtz. going to mess around with the speech recognition. thing.

gtoledo3's picture
What are you using to make

What are you using to make your models, out of curiosity?

gtoledo3's picture
Yeah, you should check out

Yeah, you should check out the .qtz loader patch... there are different ways of approaching switching between qtz's/macros... and I can't say I know the pro's and con's of each.

With this though, you wouldn't be patching multiple qtz's together... it would all rely on the image downloader refresh. If you look around here in feature requests, I'm pretty sure I posted a link to a decent web image downloader. So all it would take is setting up something to feed it url's. I mean, the ultra simple version would be to just publish that file input and let the user pick, or set up a bunch of photos in the thing with a multiplexer and make it very unchangeable... but I think it would be cooler to do the url's via an RSS and make it slick.

If I could figure out how to make a panoramic video clip (any hints anyone) it is ALL over. I would never get my head out of this computer. As it is, by looking at panoramas, I have figured out some ways to "animate" the background just by using a folder images macro type of setup at the right "speed"... but that is more of a for render thing since it can be horribly slow with high res pics, and also clunky when try to do speedy image changes (30fps folder images and the computer groans unless it is strictly for render)...

Here is a clip of a "pseudo" animation technique for the skin of a md2. I will have to render one of the things I am talking about with animated panorama background, but when I am happier with it. I think you can see how if you got a panorama and started riffing off of it in photoshop (or Gimp), made a flipbook type image sequence... that all of a sudden you would have a living background as well.

dust's picture
modeling

i use maya, and zbrush if i want to do realistic things, also i use smith micro poser pro if i want to a humanoid, cause i find the cloth and hair dynamic stuff much faster than maya, once i get a good body off the smith micro developer body, i'll send the face texture map over to zbrush and use the photo texture to add details like wrinkles and skin potholes etc. then send it over to maya to re animate. if i could get maya fusion to work with smith micro i could make my body, take a few pics of my head wrap around a model, make some quick clothes in maya as regular obj models import them to smith micro clothifie them then save the file and open in maya and have all the dynamic controls in maya, but i used hex fiend to install the program and the fusion plugin dosen't recognize my hex fiend. so i got to long way, export objs with bones and stuf then do the weight painting in maya to animate. today i messing with real flow and going to throw that into quartz in a bit after dinner. i got houdini but haven't put in much time to learn it. i really want to learn open gl get this maya 2 md2 plugin if i can find it and make some games for my ipod.

gtoledo3's picture
Thanks, that was a really

Thanks, that was a really informative rundown... I have not used anything besides Blender, Sketchup, and then various file exporters at times (well, I have briefly used metasequoia). My experience in that is very limited.

Though, I have to say that the thing you said about the ipod...on a related note, it seems like I saw some kind of opensource library for arcade classics (the blocky pixel stuff). That would be an awesome qtz/plug-in/clip collection ... I guess there would be a number of ways of approaching that.

Hmm, maya to md2? Interesting...

dust's picture
pano video

well a panavision camera unfortunately can not be bought, which has some wicked large format lenses or what ever, but that film for video you can just get some sort of panoramic fish eye lens or what ever, even then you have use two angles and stich. http://www.ptgui.com/ is some software made for 360 panos. i bet if you do some research and invest a grip of money in camera stuff you can do it. i suggest the DIY solution. i have been thinking about this for a few weeks now.

i have been waiting for geo browsing to come to my city. you know that gps street level view that google has like google earth. well im tired of waiting. i just got a arduino and apparently the have little HD spy cameras for 10 bucks that you can attach to the thing so i want to get a bunch of those and build a little 3d camera, not like matrix where the cameras are pointed inward but outward and video not still. then i was thinking about tagging gps as i save images but gps is only accurate to x feet so to save work im going to tag by camera # and frame number. so i will have images of 360 all around me.

then make a little program that figures out quickest distance. probably going to start with a building so from room # 400 to room #101 and the computer counts frames and determines the least amount of frames to decide a route then displays a virtual map of the route as if you where walking down the hall.

the tricky part is that i cant tag lets say a 4 cam set up as front back left right. it all depends on which direction you are going, or coming from, so im thinking of doing a sort of binary search and start in the middle once i get all the footage. im hoping that the cams will all line up with each other cause they are spy cams so they can get pretty much on top of each other.

but if you take that principle you could probably make a 360 cam for under 100 bucks, 50 for a micro controller and another 50 for spy cams that are 1024 x 768 format.

otherwise you would have to rig up two cams with some expensive lens do some funky geometry focal distance parallax math even then you will most likely have to stich.

or you could do the old take a panoramic shot mirror it and wrap in a sphere.

i get the idea your taking about with the multiple pictures as textures. im actually going to be doing that here in a few mins, with not only the textures but the model itself. not sure how quartz will handle but im hoping to everything syncs up. even better hoping i don't have to do that and encapsulate all my meshes and textures in one file. im crossing my fingers will find out in a bit. going to eat some pizza then give it a try.

TheRandomDude's picture
Don't forget cubic

Don't forget cubic panoramas, we could seriously do all this stuff a shot from the left, right, up, down, front, bottom, then stitch them together using photoshop for a lot less than a pano mirror or lens.

Just switch out the sphere for a cube and you're good to go.

gtoledo3's picture
You should check out Steve

You should check out Steve Mokris's video seminar series on Vimeo... he is obviously all over that idea!

It was suprising to see, because when I stumbled on that thread awhile back where Chris had posted the panorama example.... it gave me a totally visceral reaction!!! It was like "WOW"... I had been using the Kineme Alpha 9, and I was thinking "great, now I can easily do backgrounds"....

Seeing Steve's video seminar, it is now obvious that they had already been on that exact idea... but WAY further down that road than me, or any of the discussion that has gone on here actually. There is a really great breakdown of the whole technology, and some things that Steve is doing, like actually using progressive panoramic scans to extrapolate 3D models of the surrounding environment. It really makes all the stuff here about panorama seem quite quaint.

TheRandomDude's picture
Indeed, well lets all keep

Indeed, well lets all keep in mind that Myst has been doing this stuff for years :D

gtoledo3's picture
AH cool background on that.

AH cool background on that. See, I don't really play video games at all, so that aspect of all of this is kind of lost on me... I'm coming at this from somewhat of a pop-art perspective. Whatever that Mario Brothers game for the snes was, was like the last video game I was into. I was into all of those side scrollers, stuff like megaman, goonies 2, altered beast, contra, super mario, and actually really disliked when all of that first person shooter stuff came in like James Bond, Wolfenstein, etc... I liked the light hearted 2D stuff :o)

dust's picture
steve's camera

so i watched steve's seminar, wish i went to that school. my school may have some multi million dollar computers and rendering clusters but they are for calculating the great news i call it "global warming" snoor. its cool to see quartz being worked into a curriculum. as far as the new media is concerned at my school they pretty much have max and processing stuck up their buts, thankfully i have been able to avoid those classes because im doing the old school thing and got to skip a bunch of pre requisites or what ever. taking a interactivity class that uses isodora, so i think i will be able to sneak in some quartz. i think i have convinced one of my teachers to start teaching something practical like cocoa and iphone developing.

so steve has that wicked camera i was talking about. its cool people are just getting into pano stuff i guess. its nothing new to me. the 360 scan or what ever. my uncle wrote software and manual for a magnetic resonance imaging machine decades ago, or well for the first production models introduced. the man who invented the thing was brilliant got a nobel prize but, the business man who sold the things was shady, my grandfather was the lead radiologist on that project, but like i said the business man was sketchy and doubled charged medicare for scans cause old people needed them, and ended up pissing off the government. kind of sucks my grandfather had to go back to europe and haven't seen him much since then. i guess my point is a 360 scan is not new technology.

i don't know if you remember meta creations or not but they pretty much started adobe. well that guy came out with a bunch of 3d stuff, like ray dream designer the first program i ever bought like in 1995. i think i have been a pirate ever since then or well till the other day i bought kineme 3d and osculator. got to start supporting people that are innovating, but meta creations did a whole bunch of stuff. one technology was meta stream, and another program was called canoma, which let you take 2d pictures and turn them into 3d models like i said in 95 or something, anyways the guy that was behind meta creations actually had a immersive room in the shape of sphere. that was based off the same technology as canoma where he would he could take a photo and turn it into a real world immersive environment. im not sure who started doing this first meta or apple but quicktime VR has been out forever. i can remember using a film camera drawing some marks on the ground taking some pics scanning the film and stitching photos together over a decade ago.

if you really want to get technical about placing photos on inside a cube or cylinder / sphere you got to go back over a century to the zoetrope and other trope devices. so needless to say it is a really old technique that has been taking a long time to catch on. i have been for years wrapping panos around a sphere to make environment maps for 3d worlds. taking 2d photos and turning them 3d is pretty much been a standard for along time. it has not been till recently that software developments in machine learning algorithms have we really been able to use 1 photo and turn it into 3d. check out andrew stuff.

http://make3d.stanford.edu/research.html

it uses a MRF. if you don't know much about markov, hidden markov chains, stochastic gradient decents, etc.. i would recommend you look into . i just started getting into "random" art and stochastic music but the markov principles are actually used for lets say google, the stock market and AI.

it would actually be a pretty cool patch to have as a quartz plugin a hidden markov chain or something. any who enjoyed watching steves seminar, looks like the kids have someone special over in ohio.