Video Buffer

Hi folks, yes, i think QC seriously lacks a video buffer. I've managed to build one with synch'ed image accumulators (one per sampled video frame) , but the performance is quite relative. (from 60 to 30 fps when using this buffer with 50 frames). A hardcoded video buffer- or even a HLSL texture shader - would definetly allow for some neat live scratching. Apple provides sample code for buffering frames and interfacing with QC, howerver i don't have coding knowledge... maybe you guys can handle this...

Frz

smokris's picture
How should it work?

Interesting.

Let me see if I'm understanding you correctly... You want to be able to sample an arbitrary number of frames from video (from either live video input or from a quicktime movie or whatever source), and then have an input that controls which of these sampled frames currently appears at the output?

Should it output discrete frames, or should it interpolate between frames (if you specify, say, frame 3.4)?

franz's picture
it should work like....

-it would be able to sample an arbitrary number of frames (whose total amount depends whether it is sampled in GPU memory or main ram, though main ram would be better as it allow for a bigger amount) -from a live input (as we can already access specific frames of a QT files with external scalable timebase ) -its output would be discrete frames controlled by an input value. That's what we call a "circular ring buffer", and you understood it the right way...

tobyspark's picture
++ video buffer

shell script, video buffer, and structure manipulation are my holy trinity. video buffer is the tricky one =]

my tuppence goes -

Should it output discrete frames, or should it interpolate between frames (if you specify, say, frame 3.4)

i'd say lets get a 'video buffer lite' working before doing more luxury things like frame interpolation.

Maybe "Gate" rather than "Length"?

definitely.

An input to define the sampling mode: circular / auto-erase

definitely.

Separating the record and play inputs would allow to record AND play at the same time....

this isn't essential, but is very very desirable

Also, how would the size of the circular buffer be determined?

also same problem when in auto-erase mode, there's got to be some kind of memory useage limit we can set for fear of killing the live machine. is there a way of limiting useage to physical vram or ram, my experiments with recording to disk as a live sampler bodge in vdmx have sucked hard.

I'd like to see a dropdown enabling the user to switch between ordinal-integer-frame-addressing and normalized-float-frame-addressing.

definitely.

cwright's picture
irony

I believe smokris and I discussed something similar to (if not exactly) this last week, did we not? I'm all for this kind of thing myself. Shouldn't be too hard once QCImage is a bit more reverse engineered.

smokris's picture
indeed

Yes, I believe we did discuss this.

I think we probably have all we need (more than we need) in the Image Downloader (Blocking) patch.. it already implements a cache, which could be transformed into a sample buffer.

(Unless we want to implement interpolation, in which case we have a lot to figure out.)

smokris's picture
GLSL / CoreImage

Do GLSL / CoreImage even support stateful behavior? (i.e., can we use 'static' variables, or have a global array to store images? last i heard they only support stateless operation based on specified inputs to the function.)

So I think this would have to happen in Objective C.

cwright's picture
Starting off

Ok, I'm starting to work on this patch a bit. What should the interface be like?

Obviously, there should be an Image input and output. I was thinking of a Length and an Index input. The length would control the circumference of the circular buffer, and the index would address the image relative to the current image index.

This doesn't seem like it would allow much power though, since only 1 frame is available as an output at any given time.

An alternative would be a structure with all the images in the buffer. This would make DSP-like addressing possible, and I can't really think of any downsides.

So I'm partial to the latter, but maybe someone has better usage ideas.

smokris's picture
Maybe "Gate" rather than "Length"?

I'd suggest maybe using a "Gate" input that would both trigger sample-start and would implicitly set the length of the sample.. I anticipate this being used in a live situation where you wouldn't necessarily know how long you want to sample beforehand.

franz's picture
Controls

  • An image input
  • An input that would trigger the "record" mode, and that records as long as the input is set to "true"
  • An input to define the max length of the buffer: 100 meaning 100 frames....
  • An input to playback the sampled frames: - an index number would need to be linked to a scalable patch time to be able to scratch
  • An input to define the sampling mode:

    • circular means that the buffer never gets emptied, and that you write images over existing ones, and add new samples in the total buffer
    • auto-erase means that every time you restart sampling, the buffer is reset to 0, meaning that you can sample only 10 frames for instance and loop them.
  • An image output, of course

  • A current index output, for monitoring purpose
  • A Length output, that gives the total number of sampled images actually stored in the buffer

Separating the record and play inputs would allow to record AND play at the same time....

edited: Outputting a structure containing all the frames seems like a brilliant idea! Imagine the Itunes commercial (Core Animation based, the one that transforms CD labels into 3D city...) based on the structure output by that buffer.... Btw, does the structure class in QC allows to be filled with images ? I thought it was only suited for strings...

cwright's picture
Structure

As far as I can tell, you can put any object into a structure. That functionality may not be used in Quartz Yet, but there's no reason why it can't work (that I know of :)

Scaled-time playback seems cool too (negative scale coefficients woudl play backwards, etc). Buffers would need to be pretty big for this, but that shouldn't be too hard.

franz's picture
buffer size

a 100 frames buffer would be a good start - 4 seconds of video. Scaled-time input (for recording AND playback) would also allow time-lapsed recordings... like one frame every 10 seconds....

cwright's picture
gate works

I like the idea of using "gate" and the non-explicit buffer size.

What's a useful way to get data out though? Is structure good enough, or should there be a scaled playback input too? If so, how does it know where to start? The first sampled frame? How can/should this behavior be overridden?

franz's picture
Data type

Honestly, i don't know if a structure would be good enough, as structures are very under-developped type under QC. Structure has the advantage of outputing all frames at once, which is a pretty neat idea, however, how does that load QC (in terms of performance), compared to outputting a single frame ... Structure would allow, for example, to display all frames as thumbnails into one big picture... But a scaled playback input would be really nice too, as handling timings "by hand" in QC is quite a pain and not so reliable (currently, i connect a "patch time" to a scalable javascript patch, in order to generate my timecode). So i imagine i would do the same if the buffer outputs a structure (connecting the scalable patch time to the index of the structure). But wouldn't it be more convenient to connect the scalable patch time directly to the buffer node... (because less node, less wires, less mess) ? As for the first output frame, this should depend on the bufffer mode: if straight mode (auto erase style), it should obviously start with the first SAMPLED frame if circular mode (ring buffer style), it could start with any sampled frame... Finally, i would say that if it is easier to start coding the output as a structure, let's try this and see how is the performance...

/ another idea: the output could be controlled by a float, instead of a scalable patch time, allowing for easy scratching: 0 is the first frame, 1 the last, no matter the total number of frames (this is also so kind of scaling though)

/ maybe 2 behaviours avaiable: output as image and output as structure ?

edited: anyway, going the structure way would require additional "structure manipulation patches", such as "replace data @index", "merge structure"...(in a way as VVVV does with slices, for those who already used that brilliant app)

cwright's picture
structure performance

Regarding structure performance, I imagine the impact is pretty small. I'll make some tests to see though. I agree that the structure type is very underdeveloped, unfortunately. (more feature requests for that I guess :)

I still Love the time scaling control. I'm wondering if exporting the buffer as a structure, as well as having a scaled image output, would be overwhelming? Probably not, it's only a couple ports.

Normalized scaling is brilliant! so 0 would always be current frame, and 1 would always be the end of the buffer. In that case, the number of recorded frames as an output would be handy too.

How would the user control the initial frame with circular mode? It could start with the oldest pretty easily, but being able to start at an arbitrary index would be handy too I imagine. Also, how would the size of the circular buffer be determined?

smokris's picture
frame addressing

I'd like to see a dropdown enabling the user to switch between ordinal-integer-frame-addressing and normalized-float-frame-addressing.

DanieleCiabba's picture
queue?

I don't know if it will seem superficial, but have you tried with the queue object(leopard only)?

PreviewAttachmentSize
buffer1.qtz6.11 KB

cwright's picture
Back in my day...

This was more of a need with Tiger, where there wasn't a way to buffer video well. The eventual solution was the Structure Tools patch (which was only ever a Beta, before Kineme.net started its beta program).

Now that we have Leopard, this isn't so much of a need anymore. :)

marsman's picture
Structure tools patch

Since I'm still running Tiger (don't want to upgrade at this moment, that will come very soon) I would love to see this Structure tools. Furthermore, I don't seem to be able to bind an inputImage to an outputImage. What I naively tried to do is set the pointer from outputImage to the same address as inputImage using the Tiger QC template, still I don't get the patch working. Any tips?

cwright's picture
Structure Tools + more

you can find a beta of Structure Tools at http://softpixel.com/~cwright/programming/Quartz/ (the old beta page)

Can you explain a bit more about the image stuff? Are you trying to pass through an image?