best approach to video scratch plugin.

dust's picture

my primary question is :

whats the best approach to scratching video's in quartz composer ?

below is a synopsis of method i am using:

it seems in order to scrub the movie patch in async mode you would need to set a new start position and restart the player to make any change to the position. av foundation on the ipad is doing a similar process to just change the volume of video file.

so i have sought out to find a better solution to this problem. realizing that the movie player patch was just not going to do what i wanted it to do in async mode i decided to turn it of. then starting with just the most basic parameters i figured i would split the audio from video file and deal with that separately.

by using an audio unit component i'm able to graph out a 3d eq bus mixer and pitch effect pretty easily. core audio seems perfect as a host because instrument nodes can be patched in later.

i figured having direct access to the audio buffer was going to let me be able to change rate and pitch etc.. turns out apple provides some nice pitch and time units that let me easily change the tempo with pitch and key lock so that was nice to find out.

i decided to use the audio buffer to get peak amplitude and spectrum data to draw the audio waveform, plus calculate bpm and sync the video at the same external time.

i did this by using nyquests theorem to translate my audio graph context into regular time units that qc uses. yes i know that sounds like fancy math but its really just taking the audio buffer length and dividing it by the sample rate..... getting the spectrum requires the fancy fft math ;)

i was wondering what approaches other people have taken to scratching video? what the best formats are etc... i'm using a mp4 rip from youtube to test with. i have tried to take into consideration the emulation of a turn table by adding opposite forces to the back scrub.

the video is me testing the accuracy. i like the idea of using the video its self to scrub like a kaos pad. in the vimeo video i show the x scratch feature. y will have a video effect.

laserpilot's picture
Re: best approach to video scratch plugin.

May not apply because I work mostly in jitter with this kind of stuff and you're asking higher level programming questions, but I've found wacom tablets to be awesome for scrubbing. They have super high resolution so that you can get individual frames. I even added an ability that if I hold the modifier button on the pen then it zooms in on the timeline the currently selected frame so that I can have even smaller accuracy...

I've also understood photojpeg/motion jpeg to be the best for jumping around in..since there isn't as much data compression going on as in an mp4 or h264

dust's picture
Re: best approach to video scratch plugin.

so you can get better accuracy with a tablet ? normally in qc i find that the opposite. although i got a touch tablet and am able to hover over a port with the stylus and get a tool tip which i previously wasn't able to do.

when doing interactions apples track pad seems to work better than the tablet. in most audio programs or most programs in general it seems the high level of grid resolution accuracy you talking is not addressed. for instance a control will fly way out of normal range with just a tiny push of the the tablet, which is annoying when using one.

unfortunately i used a regular mouse for to many years and am unable to comfortably use one any longer. its funny apparently an apple mouse has more clicks in it then my arm did. so i use a tablet, it feels more natural and ergonomic to. i know there are these cool looking sideways mouses and what not now but i like a stylus... regardless if it drives other people crazy that try to use my computer. i don't see the stylus going away either look at the ipad.

yes i'm asking for a high level solution, like with quartz. jitter being a lower level solution. i had to get really down to the core with this. qc doesn't address audio like jitter does. its interesting max has all these cool msp objects for audio that you can use with qt.jit but jitter deals with audio as well.

i decided it was best to get down and dirty with audio units and components. sure yes i have to deal with raw audio buffers and have to programmatically graph out my audio unit nodes and parameters but once i have a dynamic factory in place i can deal with the audio unit components at the highest level. using the units gui view. this will be much better than what i'm doing now. creating my own gui controls for units that actually have gui's all ready.

i sort of like designing my own interface elements so i don't mind creating custom parameter controls. that is an interesting idea about pulling up a zoomed in view of the video frames. what you don't see here is that i'm generating the audio wave form as well. i'm not showing it because when i scratch the waveform draws over the previous drawing. not really ready to implement to fix it yet, but that was my intention to zoom up into a hyper audio video frame view. i like using mp4 because loading the audio file isn't an issue with audio units. otherwise i would have to split the audio with quicktime or something before i process it. or make the user do that.

pixelnoizz's picture
Re: best approach to video scratch plugin.


here is a new video from get control, this is working with tablet.

yanomano's picture
Re: best approach to video scratch plugin.

Regarding the format, if you are talking about a quicktime codec : Go for a format with an INTRAframe compression : i would say Photojpeg is a good guest because there is only spatial compression with it. Temporal compression in MP4 is very intensive to decode for the CPU because of the INTERframe compression (need to read some other frames to decode the current one) and it will probably drop some frames when scratching.

dust's picture
Re: best approach to video scratch plugin.

get control looks polished. i'm getting there still building up functions. looks like there is a buffer sequencer and some other cool things going on. kind of reminds me of vdmx aesthetically i guess. what is it made with ?

m4v and mp4 seem to work for both audio and video but yes the photo jpg qt movie scratches a lot better even if its file size is twice as big. i'm actually noticing a lot more sensitivity. to much almost i just need to get used to it now.