|
|
phoneme opcode speech synth video avatar (Composition by dust)
so not really sure what to call this. its basically the speech synth plugin with phoneme opcode output and rate input. the phoneme opcode is a code that represent a phoneme. this patch maps the phoneme opcode out to video clips of the phonemes. years ago i recorded this actor reading the opcode and chopped it up... while i was searching for cross modal connections between vision and sound. http://en.wikipedia.org/wiki/McGurk_effect turned out to be more of a max headroom type of effect i guess. simple one to one map. type in text watch video speak the text. this is done all the time with animations though normally just a few key blend shapes with various weighting can produce most lip syncs. so normally people don't actually produce 42 different shapes for each phoneme. either way you have this output that can be mapped to what ever you like with a rate input to speed up or slow down the voice. |
here is the video avatar with the needed plugin and video avatar. just run the decoder file after you installed the plugin. (edit new version attached)
• edit
i noticed this example patch's asset paths didn't have a ./ in them. i have fixed all 40 phoneme clips so this example number 2 should run in lion.
so i tweaked this out a bit to give a more smooth natural feeling. i kind of liked the glitched out stutter but doesn't really prove this example. so i'm including this revision.
in this patch i'm using the phoneme opcode as an external time base to the video clips. so now the video is playing at the same rate of speech as the synth and provides an almost transparent talking video avatar. what i mean by transparent is you cant see that this is composed of phoneme clips as the video looks natural. the synth on the other hand has come along way but i feel still needs a bit of work to make this sound more natural.
attached is a list of phoneme symbols and code. it will help if you want to make your own in 3d or something. also see the new clipped video examples you can use them as a base to draw 2d mouths if you want to do cartoon effect.
brilliant ! extremely useful, thxx
i thought this was kind of cool. it actually is based on the first computer program i wrote in 2008, with max msp and a speech external.... hence the low quality video. quartz composer really polished the effect with its external time base. my first patch kind of stuttered but it worked well enough to show the McGurk effect... http://en.wikipedia.org/wiki/McGurk_effect. i need to make a new HD version with multiple actors. its a little tedious but basically the actor just reads the phonemes from a list then chop them up.
in retrospect instead of hardwiring all the phoneme videos like in the example one could just use one video importer and a string printer to call phoneme videos all from the opcode output, seeing the name and opcode match just appending .mov to the opcode should work the same. i suppose with HD it might be better to load them all up into a multiplexor ? either way its a bit of set up but it produces a cool effect.
originally i was thinking i could use the audio from the movies somehow. (didn't really sound good) now I'm thinking it would make a cool interactive help menu type of thing ? it would be cool to see if someone else's avatar. it might even look cool if each phoneme video was a different actor maybe ?