streaming data into QC with Structure from File (File Tools)

volkerk's picture

What i try might be pretty bold, but worth it if it would work.

I try to stream data from property list files into QC with the kineme Structure from File patch. Basically just sequential named binary plist files (data_1.plist, data_2.plist, ... , data_x.plist). Each plist contains 3D coordinates for a voxel type animation. For smooth real time animation, i need to get 30 frames per second. Everything works fine, but I do not get more then ~5-10 fps update rate from the structure to file patch, whereas the composition runs in 60 fps.

Hard disk transfer speed is not the bottleneck, each plist file is only 140kB, which is about 4.2 MB/s total. It seems the Structure from File patch does need more then one cycle to read the data and shoveling it into a structure and to make it accessible to following patches. The performance inspector indicates that the structure from file patch requires ~53% of the time and an average of 0.02 sec .

anybody any idea why?

cwright's picture
Re: streaming data into QC with Structure from File (File ...

Loading a structure from disk is simple code-wise, but expensive in terms of CPU time.

I'm guessing that it's more of a Cocoa-framework-shouldn't-be-used-like-this type issue.

The execute function is really simple, and has no loops, so it's not like our code sucks or anything.

- (BOOL)execute:(QCOpenGLContext *)context time:(double)time arguments:(NSDictionary *)arguments
{
   if([inputSignal wasUpdated] && [inputSignal booleanValue] && [[inputFile stringValue] length] > 0)
   {
      NSString *path = [inputFile stringValue];
 
      // is it an absolute path?  if so, don't do anything
      if([path characterAtIndex:0] != '/')
      {
         if([path characterAtIndex:0] != '~')
         {
            // not absolute and not ~-ified, must be composition relative
            NSDictionary *options = [self _getUserInfo:@".options"];
            NSURL *url = [options objectForKey:@"compositionURL"];
            path = [[[url path] stringByDeletingLastPathComponent] stringByAppendingPathComponent:path];
         }
         else    // it is a ~ directory, expand it
            path = [path stringByExpandingTildeInPath];
      }
 
      NSData *fileData = [[NSData alloc] initWithContentsOfFile:path];
 
      if(fileData)
      {
         NSString *error;
         NSDictionary *fileDict = [NSPropertyListSerialization 
                              propertyListFromData: fileData
                              mutabilityOption: NSPropertyListImmutable
                              format: NULL
                              errorDescription: &error];
         if(error != nil)
         {
            [outputStructure setStructureValue:nil];
            NSLog(@"Error unserializing file: %@",error);
            [error release];
         }
         else
         {
            QCStructure *structure = [[QCStructure alloc] initWithDictionary:fileDict];//[self arrayToStructure:fileArray];
            [outputStructure setStructureValue: structure];
            [structure release];
         }
      }
   }
 
   return YES;
}

As a workaround, what happens if you combine all the plists in to a single large plist, with each sub-list being an element in an array. Then, to switch structures you simply change which index you're using. Then you're only loading once.

(And, of course, as I look over the pasted code, I notice a memory leak, so repeated loading is a Bad Idea. Fixed for the next version though...)

volkerk's picture
Re: streaming data into QC with Structure from File (File ...

thanks for the reply I tried the method of combining all data in one large plist, but the apple tool (plist editor) is not capable of dealing which such large volume of data. it is 740 MB total as single binary plists. I might have to write custom code to do that. When i tried with a shorter combined plist, it was working with structure from file, but long loading times as expected. An additional boolean output at the patch, giving a pulse once the structure is created would be useful.

v.

franz's picture
Re: streaming data into QC with Structure from File (File ...

why not try with XML ? it might be a better choice to handle LARGE datasets. Anyway, a 740 Mb file will take ages to load.

QC's XML has a progress indicator, and sends a bang when finished loading.Might be usefull to sequentially load multiple files.

As far as i remember, it is also asynchronous, so loading won't choke your FPS. Normally.

volkerk's picture
Re: streaming data into QC with Structure from File (File ...

well, in xml format the data is 6.8 GB big. making it binary gets rid of a lot of overhead.