timelapse

Quick rundown of a project I've had running at work for 8 months or so, based on QC. It's a composition (well, 2) that produces a timelapse video of a construction site where we're building a new college campus:

Dunno why, but youtube has messed up the encoding quite badly. I did try vimeo but the upload ground to a halt each time :( Also, the quality at the start is terrible - the camera has no sun filter until later on so lighting is horrible, and my composition slowly improved as the building went up ;)

We have one really, really bad quality webcam pointing at the site. It doesn't do image capture as such, but works as a network camera you can connect to through a browser, then streams an mp4 which I've hooked into with QC.

The camera focuses poorly, doesn't handle wide dynamic ranges well at all (it's pointing south and the sun crosses the screen.. ow!) and the sensor is rubbish (640x480 max). The picture I get is blurry with lots of noise, especially early or late in the day when the light levels are low. Worse: some idiot decided home broadband will work fine because it's "20MB/s speed" - forgetting that it's 20MB down, only 0.5MB up, and the video is sent up the link. So the bitrate is really low too :( Well, I do like a challenge :D

I already have a bunch of stuff designed to capture images from worse situations than even this, so I figured it shouldn't be too hard. So here's roughly how I handled it:

• I split the application up into 2 parts. The first pulls in the video feed, processes the raw image data to clean it up a bit, and spits out a jpeg every 60 seconds. The 2nd part pulls in these images, and processes them into frames for the final timelapse video.

The main capture app:

• It basically does the equivalent of a 60 second exposure. I average all frames received during 60 seconds. This removes all compression artifacts and video noise (it leaves 'hot pixel' noise caused by single pixels on the sensor which are visible still, I could remove these too with 'dark frame' removal but decided not to bother). The result is a fairly clean image, but it's still blurry with very little detail. This also gives a cool motion blur effect on the machinery moving around and limits 'strobing' type effects (when the sun goes in and out behind lots of small clouds on a timelapse video, the result is a strobe :( )

• I then process it. The main part is a custom sharpen filter. It's pretty much a standard deconvolution, but I've adapted it so I can crank the sharpening up really high without the artefacts that usually causes. I probably get 8x more sharpening out of this little beast, it makes a lot of the detail visible :D

• The final image is just dumped out to disk as a jpeg. I have nearly 300,000 images so far! The folder crashed finder regularly, and terminal's cp/mv commands can't deal with this amount of files, this gave me some headaches ;) I've now got it split into month-sized sections.

This part of the comp has been running since January, 24/7, and has been rock solid (well, I give my mac a reboot once a week.. but I'm also running a whole ton of other stuff on the same box, including windows under vmware, and it's only an old imac :) I'm really impressed with how well QC coped with this :) Especially as it's handling a live video stream, and all of the processing is done in 32bit mode (the exposure part totally needs it).

The one thing I found really hard here was 'error handling'. If the video stream drops out (which it does, very regularly!) I have to detect that, 'pause' the rest of the comp, and wait for it to resume. Sometimes it doesn't resume, so I have a 2nd layer of protection that resets the movie loader patch every 30 seconds if the stream fails, in the hope of restarting it. It took a few goes to get this part right :/

The 2nd part:

• I use kineme's excellent 'directory scanner' patch to pull in a list of all the files. I was pretty amazed to see it produce a structure of a few hundred thousand file paths, even if it did take nearly 60 seconds :D

• Next, I step through this massive structure, loading one image per frame.

• I've got a filter and a bit of logic that detects if the scene is dark, and rejects these frames. This removes night completely, which is why it never goes dark in the video :)

• I've got a nice javascript patch that reads the filename (the filename for each image is a timestamp from when it was created), and detects if it was taken on a saturday or sunday. Weekends are also removed, so it should be constant action.

• Now I do more filtering. I use the area minimum, maximum and median (or maybe average, can't remember) patches to get vec3s holding the min, max and medium colours. This drives a CI filter that equalises the images - it evens out the lighting and reduces flickering a lot.

• Again, I use my exposure macro from the first comp to collect a number of images into a single output frame, to get decent motion blur etc. I have some logic that determines the exposure time based on the number of source images and the desired video length. This is actually hard, because I'm removing an unknown number of weekend and nighttime shots.. well, I picked 120 seconds for this video and it came out at 103. Not too bad.

• Last I apply a little motion blur type effect, combining the last 5 frames to smooth it a little more and write to disk.

There's probably 6 more months of construction to go yet, but the camera is down so I've just finished the 2nd part of the project off. I'll update the video at the end.

If anyone wants more details or anything from my work let me know. I won't post the .qtz files, they're a) a mess, b) hard wired for this particular case and c) owned by my employer. But I'm happy to share any of the parts (most came from my own projects outside of work anyway).