Structure Tools Discussion

cwright's picture

There has been some recent discussion regarding the Structure Tools patch and its functionality.

From franz:

Actually the blender doesn't seem to blend images.
However, this still being a kewl option, i think you might merge "blend" and "play" patch into one...
As the "play structure" doesn't allow to speed up the playrate or to scracth, whereas the "blend structure" patch allows these.
Or keep both patches, allow blending for both, and have one that is
  • index based player, with normalised index option (as is the actual "structure blend" patch)
  • speed based player, with a scalable speed input (as could be the actual "structure play" patch, expect that is plays @1:1 only)

Then, from tobyspark

at the moment, i want to use the tools in two basic modes, one is a sampler, and one is a delay.
  • sampler would clear the buffer on the gate going high, and then continue to sample until either the gate went low or the max buffer size was reached.
  • delay would continuously sample and allow me to scrub back in time.
these are the issues i have with the tools as-is. see the two attached qc patches, you can replace the vdmx input node with a video input one to test.
sampler - i can put a 'signal on rising edge' qc macro on the clear input so that when the gate goes high, the sample always starts afresh. what i can't do is stop it from sampling when it gets to the max buffer size, as it rings around.
delay - i have implemented this by recording constantly and patching the index output to the playback index input of the blend node, with a delay parameter added to it. the patch seems to work, but i get playback glitches: if i leave the delay parameter alone, it should just be the live video straight through, but i see the occasional glimpse of an older sample.
i've looked at the code, its quite readable and the comments make sense. so i'm looking to have a poke at the record node later to see if i can implement a mode for it that directly matches my sampler requirements. the delay one i'm no so sure about, short of writing a new node that is actually a continuously sampling fifo, shifting the frames down the index. before i go any further how does that chime with your use/ideas?

The FIFO mode toby described was actually default behavior in the initial beta. I'll add the option for that functionality, as well as "clear-on-gate" functionality from an input.

Franz responded with the following:

I think Chris should include different options to set:
  • ring buffer or slide buffer
  • stop at end or loop (even ping-pong could create interesting results)
What do you think guys ?

So, other than the drop-down functionality, what needs to be added/modified? As franz noted, the blender doesn't blend yet, but that is in the plan for the future. I'm not sure if the player should do blending — we need some way to get unmodified data out of the structure just as it went in. But there could be an option to just output data unmodified, so I don't know what the "Right Thing" to do is for this.

See also VideoBuffer for an early discussion of basic functionality.

tobyspark's picture
spark spec

from the discussion we had, and looking at it again now, i'd suggest two nodes, a "record lite" and a "record pro". the "record lite" would serve the common cases of a delay and a single sample record. the "record pro" would be fully manual. and of course, the "record lite" could just be a macro of "record pro" set up right. "record lite" in - object in in - sample or delay in - gate in - max size out - object structure out - structure size where delay gas gate without clear + slide addressing and sample has gate with clear, one-shot addressing, stops when gate low or max size reached. outputs sample size. "record pro" in - object in in - gate without clear in - gate with clear in - manual clear in - slide (ie dsp) / one-shot (ie as-now with stop on end point) / loop (ie as now with loop on end point) addressing in - 'record in' index (where it starts recording on loop or gate high) in - 'record out' index in - max buffer size out - object structure out - structure size out - current index

cwright's picture
Record out?

What is the purpose of the "record out" input? I'm guessing it has something to do with the structure's initial output, but I'm not sure? if so, that functionality can be achieved with the blender.

I think I'll make the "record pro" defaults behave like "record lite" so that we won't have two patches floating around. 7 inputs and 3 outputs isn't that bad really. :)

tobyspark's picture
with record in and record

with record in and record out i was thinking how a sample buffer could be (mis)used beyond the simple one-shot buffer.

if you had in and out points, then you can basically edit a timeline together. which is something i could definitely see myself doing, making glitch/rhythm cuts on the fly.

tobyspark's picture
'record lite' node

I think I'll make the "record pro" defaults behave like "record lite" so that we won't have two patches floating around

...well the reason for having the record lite was that there are two, different, sets of defaults you'd need depending on whether you're making a sampler or delay(er). but yep point being once record pro exists then two macros can be trivially made to give the two record lite 'nodes'.

smokris's picture
not macro but friend classes

I'd suggest making the 'lite' patches as first-class patches (as opposed to macro patches), so that we don't have to also distribute a .qtz clip along with the .plugin file. The 'lite' patches could be pretty easily constructed by making some friend classes of the 'pro' patch, which just forward most of the messages on to the pro patch.

cwright's picture
endpoint?

So if record in is specified, it determines where recording begins. record in = 5 means first sample goes in 5.

With record out, that determines the stop point? That's what isn't clear to me :( if record out was 10, that means recording stops at 10?

cwright's picture
2 it is

Ok, 2 patches it is :) Should we keep "pro" and "lite" as the nomenclature, or should we pick something else? ("basic" and "advanced"?)

tobyspark's picture
pro and lite are awful

pro and lite are awful names, its just what came to mind at the time. basic and advanced is much better =]

tobyspark's picture
yes, sorry for not being

yes, sorry for not being explicit.

these are in- and out- points, specified as structure index integers. so record in is the index to start at when gate goes high (by default 0), and record out, if non-zero, greater than the record in value, and less than the max_size value, is where the recording stops/loops.

tobyspark's picture
oh - strike 'out - structure count'

...as there is the built-in structure count node if needed.

cwright's picture
non-zero record in on first pass

How do we deal with a non-zero "record in" when the buffer is empty? Silently fall back to zero-indexing?

tobyspark's picture
hmm.

good question. not sure.

i think its bad for a qc node to silently do something different. i'm not sure what the consequences are to fill the structure up with null objects if required.

if we also had the suggested elsewhere set of nodes to manipulate structures, we wouldn't need the in- and out- points, as the more avant garde uses i was thinking of could be achieved through merging structures.

franz's picture
Why 2 patches ?

Hi, sorry to jump late in the thread, but why 2 patches, that do the same thing ?. Basically, one big patch with all the options avaiable , is, imho, nicer , it is less hassle..... and one help file or in)-patch description well conceived, would prevent you for making 2 versions of the same thing. To me, the simpler the better.... (i can give a hand for the help section if needed)

EDITED/ -One patch to determine how frames are stored into the buffer (with all options settable: stopmotion,DSP mode, cyclic, stop @end......) -One patch to determine how frames are played back (with scalable timebase, forward backward.... ) -One patch to manipulate structures (sort, merge, clamp(=in out)...)

franz's picture
keep it simple

yes, keep it simple. I think in- out- points is something that can be easily sorted with: -structure count patch (dynamic) and some sample and hold (hidden) patches..... There's no need for a specific patch with these in)- out options, and if you're making a set of tools to manipulate structures anyway, we'll have all at our disposal to work correctly....

tobyspark's picture
my thinking was...

haha - two patches because most of the people out there need the basic patch, and we'll be wanting the one with a gazillion options. so lets have both =]

documentation - the basic patch should document and largely support itself, by only giving a clear choice of 'sample' or 'delay' functions. the advanced patch will be used by people who should be able to work it out, or know why the particular combination of clear on gate with x and y isn't so useful.

technically - its still one codebase due to 'friend' classes.

but most fundamentally, its keeping it simple for all users, us included, by having the basic patch: it does just what you need most of the time (if you don't think it will, holler quick). the advanced patch is just exposing the functionality that the simplified one uses behind the scenes.

those are some reasons that got me there, but hey, its just a view.

cwright's picture
Advanced Beta 2

Ok, I've created Beta 2, using much of the feedback from this discussion. Only the recorder is changed from the previous beta — player and blender are the same. Available via the usual place.

There's a stub Structure Record Basic, it currently doesn't do anything (I think it's an old copy of the Advanced one, so it might do something. Don't Use It :).

This version uses CIImages instead of CGImages, so it might be a bit faster (and possibly a bit less stable, I haven't put any error checking in yet). Please let me know how this goes, especially on non-Intel GMA graphics cards.

franz's picture
excellent

thanks a lot, this is top and working as expected. I guess to have a delay, just switch to DSP mode and select the corresponding index (index 30 for 30 frames delay).... you don't even need the player patch. about "structure play" patch: instead of rate divider, could you implement a "rate mutiplier" so that >1 values speed up the replay and <1 values slow it down..... (and that accepts floats...hehe)... And i didn't notice any dents on stability so far....

tobyspark's picture
performance excellent

the performance is excellent, between using CIImage and having the delay functionality directly in the patch everything is now perfectly fluid. huzzah.

there is a bug is structure blend: when set to normalised and repeat, an input index value of 1 corresponds to outputting structure index 0. clamping gets round this.

structure record functionality + anal semantics - still doesn't have the key feature i desire, which is the ability to simply make a make a recording of arbitrary length limited by the max sample size. i don't want it to loop, and suggested there was an input choice to manage looping or stopping when it reaches the max size. - the record mode names aren't too clear. what about "record when true", "record to max on signal", "sample on signal" (in qc parlance signal = on rising edge). - now we have the different record modes, "gate" would make more sense to be "record" - "buffer size" is more accurately "max buffer size" or even more accurately max number of samples

structure record bug - what i can't explain is that when recording a 'image with movie' patch, everything is fine. when recording the 'video input' set to the built in isight, every image of the structure is the last sampled frame. but having said that, my first use in making a delay did work using that. will revisit this later with a clear head and after a reboot.

cwright's picture
Notes

I think the video input effect might have something to do with me using CIImage incorrectly? I'm not entirely sure what causes it though. If you figure anything out, please let me know :)

Thanks for noting the Blend bug, I'll try to fix that (I know the cause, 0 is congruent to 1 (mod 1)). I'll try to whip up a non-hack fix in Beta 3.

I agree on the input and mode name changes. I'll work that in too. Don't think up descriptions when you're tired, it comes out in Geek :)

I'll add a "Stop on end" mode too, is that the correct functionality of the feature you noted?

(I'll be away from my desk till Sunday, sorry for the impending delay)

cwright's picture
player scaler

I'll try to implement the scaler in the player; it shouldn't be too hard. Negatives will play backwards too :)

cwright's picture
CIImage strangeness

Ok, I've somewhat figured out the CIImage video input strangeness. It's basically inconsistency in how CIImage actually buffers images. Sometimes it actually duplicates the data, but sometimes it just keeps a pointer to the video ram.

Basically, any Core Image filter used (addition blend mode, subtraction blend mode, et al.) will get incorrect data (pointer to live), while patches that convert it to some other format inexplicably get the correct (buffered) data.

Fun trick: video input -> structure [dsp mode, buffer size = 2] -> item at index 0 and item at index 1, pass both to subtraction blend mode patch -> billboard. This Should be a basic motion detector (difference of adjacent frames). However, it will always be black because the inputs will be identical, and x - x = 0. The weird part: Mouse over the subtract blend mode's output image, and the tooltip will have The Subtracted Image in the thumbnail view. This probably happens because the tooltip viewer renders using a sightly different pipe line to get different values for the inputs (as opposed to using the raw input data and magically generating correct results without doing anything differently).

The solution to all of this is to use accumulators. I'm working on adding this to Beta 3 as well.

tobyspark's picture
using beta 2 for text animation - huzzah!

structure record and the sample on rising edge is the key i needed to allow sophisticated text animation. spent yesterday investigating this, it works, a real breakthrough!

cwright's picture
CIImageAccumulators

Ok, got CIImageAccumulators working, performance is quite nice on my machine. I'm going to work on adding the requested features now, and hopefully have Beta3 out sometime this week :)

tobyspark's picture
good to hear + feature request

thats good to hear.

here's a feature request to come out of my text animation work. in this particular case, i want to have a structure of constant size that i can 'clear' by filling with an object. recording in then overwrites that object with the sampled one.

ie - - a 'clear or reset to max size' mode - a 'reset object' input

...this advanced node is getting, ah, "advanced" =]

if that is too much, i might be able to fork the finished advanced node to have this functionality. or, i really push my fledgling knowledge and try and create a specialised node that would be called something like 'imagestructure from string'.

cwright's picture
reset to object

I think the reset object can just be pulled from virtual in? Other than that, this would be a handy mode for creating a "empty" buffer that has a non-nil value.

So, 1 more input, clear-to-object ?

cwright's picture
Beta 3

Ok, beta 3 is out. It uses image accumulators, so it's hopefully as fast as the CIImage port, but should work with CI filters as well.

Player was enhanced with a few new features, including franz's scaler idea.

The blender wrap bug should be fixed.

New to recorder: text improvements, 1 new record mode (stop when full), Clear With Object input (to fill the buffer with a preset value).

Please let me know how it goes :)

tobyspark's picture
supercool... tomorrow to test

bedtime now, but will be testing that tomorrow.

tobyspark's picture
sampling structures

bug: using the boinx 'set structure member' i built a little structure of three numbers. the structure goes into the record node fine. coming out of the record node, i have a list of structures as expected, but each structure seems empty.

also from my look at the original beta's code, i can also imagine that it probably isn't a good idea to mix images and structures, which i didn't do this time but somebody might try to down the line.

now it isn't essential that the record node is capable of sampling a structure, as i can revert to the original comp where i had four record nodes in parallel. but it would be nice.

cwright's picture
structures

It doesn't do anything special for non-image inputs; it just creates a copy of the data and stores it. I'll look at structure sampling to see what it's doing. (Just as a confirmation, I have also duplicated they empty structure effect you've mentioned)

What do you mean by mixing images and structures? putting both at virtual in (by changing the wire or using a multiplexer or something)? it shouldn't freak out or anything, it checks each incoming object individually.

tobyspark's picture
by mixing images and structures

by mixing structures i meant having a structure with one of its keyed objects as an image.

ie what i'm doing for the text animation is sampling for every character in a string an image of the character, its width and height, and its xposition (ie a sum of all widths so far). at the moment it works as four separate structure record nodes, but it would be elegant to roll each image,w,d,xpos into a structure representing each character (achievable by boinx's set structure node), and then sampling these into a structure representing the string.

nothing to lose sleep over: elegance is optional, it already works as-is.

cwright's picture
3.1

Sorry to whip out so many beta versions so quickly... Beta 3.1 adds structure sampling as well.

cwright's picture
*understands*

Ahh, I see what you mean... hmm... this will be interesting to fix. Looks like I'll have to walk the structures recursively and properly clone images... are there any "standard" patches that put images in structures?

tobyspark's picture
check you out =]

check you out =]

tobyspark's picture
not that i can see.

not that i can see. the official nodes that give structures tend to be a bunch of information around a URL. i'd be happy just to not mix the two, but then again there's the tradeoff of documenting/supporting a limitiation vs making it work.

cwright's picture
I *think* it works

I did some testing, and I think images in structures are ok. There are probably a few caveats though, but I don't know the details.

Since images are all duplicated using accumulators they can get passed through multiple structures without problems.

Try it and let me know what happens :)

tobyspark's picture
i think it does too. my text

i think it does too. my text patch with the 'elegant' structure works perfectly. full testing including the sampler/delay inside vdmx after the weekend.

cwright's picture
Record Mode Revisions

Ok, I'm looking at the spaghetti that is inside the record's execute method, and I think it could be cleaned up by rearranging some of the record mode stuff. Basically, the rearranging would make the index form a bitmap, which we could then just mask and test. This would make the code more readable, and hopefully a bit more responsive.

So, some of the record mode bits would be:

  • Record to front of buffer (as opposed to end of buffer))
  • Record on signal (as opposed to level)
  • Stop on full buffer (as opposed to repeat)
  • work in multiple of buffer size (as opposed to not working in multiples)

So, we can mix any of those modes to determine the recorder's functionality. Does this sound sane from a usability point of view?

The dropdown would have 16 entries, one for each combination.

Thoughts?

(p.s. Currently Adding actual blending for some objects to the blender ;)

smokris's picture
dropdown vs checkboxes

Could the interface be four checkboxes to toggle the bits directly, rather than an expanded 4-bit dropdown? I think this might be easier to work with, now that we've got sixteen permutations.

tobyspark's picture
for clarity...

yep, the controls are still somewhat opaque.

my main thing would be to demarcate the inputs between mode settings and the actual business end such as record/clear/in.

i found it confusing that the 'clear with virtual in' wasn't a mode setting but a different trigger to the clear input. don't think it matters too much whether its a mode or input, but it needs to be named and grouped amongst the inputs accordingly.

i'm largely agnostic as to a index drop-down or four boolean inputs, if i had to fall one way or the other i'd go for the drop down so the record node stays a reasonable size on the canvas.

toby

cwright's picture
regarding clear

The awkwardness of the clear stuff is what made me start thinking about record stuff as well.

Should Clear Mode be an input as well, to clean up Clear With Input and Clear on Gate? Are there other clear modes that might be handy?

franz's picture
total crash

i've got a total crash when playing with "structure player" rate. Do you want the crash log ? Shall i send it somewhere ?

cwright's picture
Sure

if you could, just e-mail it to me. Any details on what the inputs were?

franz's picture
sent...

by the way, what happened to the DSP recording mode ?

EDITED: okay, the name changed to record on front of buffer...

EDITED, again: this worked, but after 5 minutes, when being at some index different than 0, it stops working, then, after some more more minutes, it crashes....

Dunno why, but this new beta seems very Unstable... Maybe it's the fact that you switched to CIimages....

cwright's picture
Crashes

From the log, it's actually crashing in QCBillboard. This is the bug I mailed you about; I'm pretty certain it has to do with CIImages as well (though I can't track down the issue). The problem's in the recorder, not the player.

Thank you for spotting this too, so I know it isn't a weird fluke on my end :)

EDIT: [by crashing in QCBillboard, I don't mean to say that there's a bug in QCBillboard; rather I'm saying that my bug triggers an error in QCBillboard.]

cwright's picture
Beta 3.2

Thanks to franz for duplicating a bug, I've managed to fix (i hope!) CIImages to not crash randomly. Blending for colors and numbers is also thrown in (not images yet though, still working out the details for CIFilter to provide hardware accelerated image blends :)

please tell me how it goes.

cwright's picture
just kidding

seems to still be there... sorry :( the hunt continues...

cwright's picture
Investigation

The problem seems to lie entirely on the use of CIImageAccumulators, not in CIImages themselves. Does anyone know of any opensource projects that use ImageAccumulators, so I can see if I'm missing something obvious?

So, until then I guess we'll use Beta 3.3, which uses a hybrid CGImage for input, CIImage buffer internally. This combination appears to be stable for long runs. It's not as fast at recording, but pulling images out of the buffer should still be fast since they're CIImages.

tobyspark's picture
just a quickie

to say really yes please to the record mode revisions, 'cos i just hit a situation where i needed record on signal and record to front.

am not thrashing it with video sampling at the mo, so can't say as to stability. it works fine for my text sampling uses at the mo.

cwright's picture
thrashing

Cool, I'll start working on that interface change :)

From what I've tested, it appears that the instability is only triggered with Video Input's input, not other forms of images? It also freaks out if you try to feed it input from Image From Movie (my local build has this fixed, don't worry).

cwright's picture
Shoot myself in the face

Ok, all this time I've been trying to get CIImageAccumulators to work for fast video copying. And, as always, there has been some instability when using them, which I thought was my problem.

I'd like to announce the identical bug manifest in an unpatched environment: http://lists.apple.com/archives/Quartzcomposer-dev/2007/Jun/msg00088.html

It crashes in the same function (CGLSetPBuffer), and sometime hangs (I've had this happen a couple times too).

So, where do we go from here? CIImage is not stable enough to be used here. Resort to CGImages?

cwright's picture
options

Easiest option: don't use it with Billboard (even after a long time with Direct Core Image Rendering enabled it crashes...)

This is a pretty big caveat though (are sprites a reasonable alternative?) [ moot point: sprites are unstable too; they eventually call CGLSetPBuffer too... ]

Use CGImages: lots of memory usage, not as fast.

Others?

tobyspark's picture
arse

this is beyond my expertise. but a stable but slow version is better than nothing?

tobyspark's picture
pure speculation...

here's an excerpt from a chat room with one of vdmx's developers.

mrRay oh, that's hilarious- he's having problems with the CGLSetPBuffer crash

toby *. from, say, "beta 3.2" post

mrRay that's funny, i responded to the thread on the apple lists that's linked to in the thing you just posted

toby *. aaah, yes

mrRay as far as i can tell, it's a threading issue within apple's software. i can reproduce the same problem by running vdmx with a live input for the built-in isight on a gen-1 or a gen-2 MBP (ironcially, gen-3 MBPs don't seem to have this problem) the problem seems to be that multiple threads are attempting to access the same opengl resources, which is a super-bad no-no every instance of a CGLSetPBuffer crash is accompanied by a matching CGLSetPBuffer crash on the thread spawned by the drivers for the built-in isight

toby *. mind if i post that?

mrRay as long as you also mention that it's all purely speculative, not at all

cwright's picture
I love MrRay :)

[he also provided me with some patches to the wiimote framework that I have yet to really get to :/] Thank goodness I'm not a complete moron :)

Is it limited only to iSight use? that's the only video source I've been able to crash it with. If that's the only input, I don't see a problem with just saying "Don't use iSight, or you'll probably explode your Mac" and leaving it at that. Or, is it any firewire camera? (implying a failure with firewire-level interfaces etc)? I can't test the firewire non-iSight case (no firewire camera), but maybe someone else can?

toby, thanks for being resourceful and backing me up here :)

cwright's picture
taintage

To further taint the problem: Since I'm free from the burden of figuring out what's going wrong now, I've been free to actually develop other stuff, like a Core Image accelerated frame blender :). After getting it to work successfully, I must say, Holy Cow, it's Really Nice, always 60fps even on my machine. There's no way not using Core Image is a viable solution :) Additionally, while not confirmed I believe the CGLSetPBuffer problem is isolated to only iSight input, and possibly other FireWire camera input -- hopefully I'll get to test the non-iSight camera stability within the week or so.

So I'm thinking that we can simply throw on a huge disclaimer saying don't use iSight, and maybe add a CGImage mode if you Absolutely Have to have stable iSight support. It'll be pretty slow at recording, but playback should be unaffected (I have yet to work out the logistics, but I think if we use ImageAccumulators derived from CIImages derived from CGImages (which are in turn derived from iSight's CIImages... sigh) we'll have no performance hit other than adding frames to the buffer.)

Is this a reasonable solution/compromise?

cwright's picture
interface

I'm going to do dropdown for now (it's actually only 12 options since Buffer-At-Once is mutually exclusive with Record on Signal), just to keep the number of patch inputs low. We'll see how that goes with a beta, and then change to bit inputs if it's easier. (how do we handle bit inputs for invalid conditions, like BAO/On Signal mode?)

tobyspark's picture
yep...

if there's get a final candidate uploaded, we'll take it from there. once i've updated the example patches i expect it will get much use and so testing from the vdmx community, and we'll see how vdmx passes the image input.

toby

cwright's picture
Beta 4

The usual, just released a few minutes ago. Back to Core Image, so it's not stable with iSights unless you have a really new MacBook Pro. Working on a degraded mode so that will work too.

I've not been able to get my hands on a DV cam with a firewire cable, so I've been unable to test whether or not the iSight issue is just iSight, or all firewire cameras. If someone could try this, I'd love to know how it goes. I'll try to do a brief independent test on Wednesday night.

I hope this is considerably closer to release :)

cwright's picture
Non-iSight is good to go

15 minutes of uptime with a non-iSight firewire camera. Going to give it a 8 hour burn-in though. Have power-cycled the camera while plugged in, have resized the window, have disabled the display, enabled the screen saver, have ocluded only portions of the window. Looking go so far... :)

[EDIT: almost 10 hours later, still stable. I think we can safely say it's an iSight driver bug thing.]