multi-touch music box

dust's picture

so this is a comp i have been working on. it is a pretty intensive process as i am using quartz composer, unity3d, and ableton live. in future versions im pretty sure this will have to be a multi machine set up one for rendering one for audio etc..

any thought on how a 3d osc/tuio/midi node graph type of program should work. its all experimental at the moment so i'm making it up as i go along.

the basic idea is to be able to place these nodes or units i'm calling them in chains on the screen to be able to make up graph that produces usable midi data for synthesis. the quick low down is, this patch is sending the position, rotation and scale vectors as midi control messages as well as a randomized arbitrary midi note map to ableton.

this is currently in progress but if anybody is into experimental ways of making music and wants to try this. i'm happy to supply the binary quartz builder and unity files as it seems things run smoother from the binaries.

here is a more detailed explanation from vimeo...

this is an early experimental multi-touch music project, i am working on. it is inspired by the reactable and utilizes the tuio protocol to transmit cursor data to unity via xTuio. in addition to being rendered in unity, quartz composer is being used to parse the spectrums peak amplitude for each instrument that is also being feed into the unity render engine via osc to visualize the waveform between the 3d music box nodes.

there are two types of nodes that both transmit midi note and control data to ableton live for synthesis. the two type of nodes are a synth unit and fx unit. although the nodes act as two different classes they are inherently the same objects transmitting the same data to live.

meaning the fx unit nodes also can act as a synth node and utilize its midi note data and like wise the synth unit nodes also send midi control messages. the nodes are transmitting a random note in the range of 0 to 8. this note is subsequently parsed and retransmitted to live in a particular scale. in addition music box is also sending 8 harmonic notes based on transformation logic as well as transmitting the subsequent rotation, position and local scale vectors for each node as midi control message.

so what this means is you could use a two finger pinch zoom to scale the node up which in turn can be mapped to an instruments amplitude or what ever parameter you would want scale to be used for. although this video is only showing positional usage both position and rotation vectors are sent to live as midi cc. doing a pinch zoom pivot rotation will give the xyz rotational messages to subsequently map.

the fx units are chained to the synth unit nodes via a spring joint so that the position of the fx unit can modify the location and rotation of the synth node by utilizing rigid body physics. meaning that the control messages of the synth unit can also be effected by the fx unit and vice versa. this is all experimental and there is no right configuration at the moment.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

dust's picture
Re: multi-touch music box

oops just noticed after watching this video my right sided fx unit has an extra spectrum band connected to the main output unit. thats not supposed to be there, as well as the right sided synth units color change on touch is not working as well. i have fixed that and have a newer version coming soon showing how the rotational vectors and local scaling will effect the synthesis. although these are boxes right now eventually when i get to the iconographical representation stage these will more than likely be models that representative of a synth and effect unit etc.. when i get to that part of the development process this may not be called music box as the boxes may not exist.

its kind of different to work with node graphs in 3d right now there is only one input an output but theoretically this type of graph could expand in different direction as the main output unit is only representative of a 2 channel master mix output. at the end you see the actual master unit produces sounds as well. this was to show how the units can universally be used as either instrument, fx, output or all 3.

itsthejayj's picture
Re: multi-touch music box

Sweet great stuff dust! great to see we are working on the same page as me! Wish you lived on the otherside of the pond here in the UK because I got some crazy stuff to show you (i think you know what i'm talking about ;) ).

I'll try and spend sometime this week to send you some Ableton > QC magic.

cybero's picture
Re: multi-touch music box

Impressive stuff, dust :-).

Goes to show how far QC can be extended [much like the light paint example you posted about earlier today] way beyond simple animations or screensavers.

jrs's picture
Re: multi-touch music box

great work - which bit is unity and how are you incorporating it?

shaker08's picture
Re: multi-touch music box

yeaahh amazing work !!!

shaker08's picture
Re: multi-touch music box

soo..can i try the patch in my multitouch box please??

dust's picture
Re: multi-touch music box

I haven't been abroad in some time. I was born in new england but my family as far as my grandfather moved back to Ireland. he has 24 kids 12 in the states and 12 in Dublin. so have been meaning to get across the pond. one of my teachers has been telling me about a graduate school in Scotland that has a new media graduate program. I am actually an interdisciplinary student so I wouldn't have necessarily go abroad to get a graduate degree but I have been thinking about it. Yeah I'm defiantly interested in any live qc jam things. particularly being able to load a qc file in session view. without using max would be awesome.

my email is dustin.oconnor@umit.maine.edu if you want to send me a link or something.

dust's picture
Re: multi-touch music box

well its rendered in unity hence the physics and particle bolts. quartz composer or more specifically kineme audio tools is parsing and synchronizing the audio spectrum with within a unity fixed update function, then doing some yield return call backs based on positional magnitudes. meaning the box is a toggle so the slow to stop inertia triggers the off once the box is done moving. also the transformation and collision data is being sent out which could be used to synchronize and render this in quartz which maybe be faster. sort of just using the unity physics engine with empty game objects with box colliders on them and then doing the graphical interface in quartz. I'm still working out possible solutions thats why I'm asking for suggestions.

dust's picture
Re: multi-touch music box

yes you can try this. I'm in the airport right now going to a gig in Los Angeles so can not post it right now as I want to save some battery power on my laptop. I actually have made a multi touch tuio controller for the iPad that works with this as well in unity. so if you don't have a table but want to use this multi touch you can. as well as hook it up to existing table with xtuio, a two touch can be simulated with standard mouse as well I will post this once I get to Los angeles.

dust's picture
Re: multi-touch music box

here is the latest build. i had a few min. layover. music box- ipad multi-touch tuio - osculator patches and live project file. the unity music source is 1gb so i cant upload right now but binary sending all data etc is working with xtuio etc...

http://pelican.asap.um.maine.edu/~oconnordw/index.php?option=com_content...

shaker08's picture
Re: multi-touch music box

oohhh thank u very much!!! i havent time to try it now because im working very hard this week,but i will deep testing on the next week. again THANKS!!!

shaker08's picture
Re: multi-touch music box

hi man!! bad news..app look sooo cool but no work with tuio simulator and tbeta.. :( and .qtz no open cause i need to have installed some extrage pluguin i think.. but good idea i hope still growing up!!