Audio Spectrogram

LuxAqu's picture

Ok I'm very new to Quartz Composer but I've been trying to find a way to create a visualizer that maps the octaves of sound directly to the visible spectrum. This is based on the idea that the visible spectrum is essentially one octave of electromagnetic radiation. More about that theory here: lukenimtz.com

Thanks to AudioTools I was able to get a decent resolution for the frequency spectrum, but being a noob I ran into a lot of technical problems that I didn't understand (with particles and iterators and render in image patches...) Plus I wasn't sure how to best capture the composition in action. In any case, I finally have something to show:

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

cybero's picture
Re: Audio Spectrogram

Nice bit of work and track.

Capturing the visualization, offline rendering can be done at the command line or using Quartz Crystal and Kineme Audio Tools, re-sync with sound in your video editor, you can even use GarageBand [Movie track] or iMovie for that .

If you are intent upon screen recording then iShowU or Screenium or Jing are very good bets with a lot of users voting heavily for the first two applications mentioned.

If you are using one of the above then I would advise capturing as close to lossless or with H.264 and do not record the sound at all.

You can always re-sync later and if one is seeking to record the audio too, then using SoundFlower will likely as not give you better results as will Kineme Audio Tools in regards of being able to be used to pick up sound sources run external to the composition being recorded or offline rendered.

Offline or online, the rendering of video capture results in some slow down on the way the composition is actually working when rendered live.

Unavoidable, although workarounds like offline rendering at more than single speed or recording and then increasing the speed of recorded video can help to better re-sync the resulting audio video mix.

In regards of using Kineme Audio Tools as distinct from but with Apple's stock QC audio patches, the low, mid, high stock processor patch will show that Kineme Audio Tools produce both - and + output which can be a difficulty in itself, although one can either use range or a re-calculation script or math expression patch to give the patches being fed such audio related / calculated from data the sort of values they like and to which they can properly respond.

Smooths patches will also help enormously.

The stock Audio Visualizer template makes good sense of the data it retrieves from the audio channels, although it can't deal as well with the sort of data range you'll get from the waveform. You can either choose to aggregate the spectrum range or simply select a particular range.

There are loads of possibilities, running parallel left and right channels visualizations from the peak, frequency and waveform data.

BTW, if you've got QuickTime on 10.6.x you can record the entire screen and then crop it in iMovie

P.S your site's link on your account at Kineme page doesn't point in the right direction

LuxAqu's picture
Re: Audio Spectrogram

Hey thanks for the tips.

I did end up using the iShowU HD demo (thus the watermark) to capture this while Soundflower was directing the audio of a youtube video, "Charlie P - Sometimes I Still Miss Her," was playing on in the background. I wish I had QuickTime 10.6...

I couldn't actually get the more complex things like particles and render in image working in QC so this comp is very simple graphically. On the other hand, it's very fast so it worked out pretty well capturing live with audio.

cybero's picture
Re: Audio Spectrogram

You're on Leopard then, so the particles stuff you had problems with would either be the Particles Engine or the Kineme Particle Tools?

Render in Image, first tip, place a clear at the root level of the RII, then place your macro you want to render in that RII patch, then output to , typically for me, a 2 * width Billboard or Sprite.

Set dimensions on the RII to zero width and height unless you specifically want to output to a different size than the composition would default to , [which would typically 512 * 384, unless you are setting your default for the comp or QC to another pixel size or aspect ratio].

Love that Aurora comp of yours, though I was half expecting that you had been working on the 10.6.x developers example [ use with care if you ever do upgrade to 10.6.x ].

I have to admit that going down the route of offline rendering [Quartz Crystal] and re-syncing [GarageBand or iMovie] is my preferred modus operandi, although QT screen recording and crop that in iMovie does do the job when all else fails and screen recording is possible.

Still, that requires 10.6.x.

A nice Freeware item that does a pretty good job is CaptureMe. Works in 10.4.x upwards, including 10.6.x

Nice :-)

dust's picture
Re: Audio Spectrogram

not sure if people know this or not but i just noticed the other day that the new quicktime screen records and you can select a particular audio device like soundflower for recording.

dust's picture
Re: Audio Spectrogram

here is an excerpt from james furia geo music in regards the the correlation of color and music. we discuss this a lot at school and people seem to think that the spectrums are so different its not really possible. well the math is the same for light as it is for sound sound so it is possible particularly if detune to 432 then everything seems to line up nicely. interesting the musical note b becomes IR.

Quote:
A SCIENTIFIC CONNECTION OF COLOR AND MUSIC

Throughout history many prominent people have experimented with a possible connection between color harmony and sound harmony: Aristotle (384-322 B.C.) Leonardo Da Vinci (1452-1519) Sir Isaac Newton (1601-1680) Unfortunately most of them adopted an arbitrary color scale. Even in our own century volumes have been written with no conclusive answers. The book "Colour-Music" by Adrian Bernard Klein, he states, "The note C is green." then he says, "The note C is red." The book gives several ideas of what the note C might be.

THEY ALL HAVE NO CASE UNTIL THEIR IDEAS BECOME FOUNDED IN NUMBERS.

According to the Encyclopedia Americana "Color and music should remain separate because there is NO real connection between the two." WRONG!

V = F x W

Velocity equals frequency multiplied by wavelength! Color is measured with a spectrophotometer and the answers are given in terms of wavelength. Frequency is measured in Hz. or cycles per second. Logic dictates that because of such a huge variance in these two spectrums, that it is illogical to believe that each note will result in a different color, and vice versa. More so, would be an idea of a bleed over, as if your ceiling were a color and you were a note standing in the middle of your room. The spectrum of color is much higher and covers more ground, so to speak. Because of a simple fact: The higher the frequency, the smaller the wavelength. It stands to reason that a lower light frequency, (red) will contain a larger wavelength. Thus, a pinpointed frequency will live under a blanket of color. In other words the color purple will cover all of the frequencies surrounding the note A. Remember, there is a lot of space in between two notes. (Or one half step in music)

The frequency of ultraviolet (466,560) will amaze all associated with Carl Munck's Earth grid system he calls "The Code" 466,560 divided by 216 equals 2160! How can this be U.V. / 108 is 432 also.

The following are the answers I've found

B = INFRARED

C = RED

C = RED

D = RED

D = BROWN AND ORANGE

E = YELLOW

F = GREEN

F = ULTRAMARINE BLUE

G = COBALT BLUE

G = INDIGO

A = PURPLE

A = ULTRAVIOLET

A Quote from "The Keys Of Enoch"

"The keys to the future physics are given in the cross matching of geometries and color codes which will allow the structure of chromosomes to carry consciousness across light-time zones. There's is a color code in everything."

cybero's picture
Re: Audio Spectrogram

Stumbled upon this. Colour music system being controlled by live frequencies, tuned to the equal tempered scale, via a single microphone [by chromogram] .

VISIC - A Proposal for a True Color Music. I like the general concept and it makes for interesting reading.

Quote:

While I have not actually implimented VISIC on a computer-midi system (lacking the equipment and the computer know-how ....I offer the idea as a prolegomenon to such an operational system

Robert Emmett Mueller [VISIC: A Scoreable Keyboard Color Music]

gtoledo3's picture
Re: Audio Spectrogram

People tend to associate colors with various emotional states. There are many studies on this, and I'm sure that a "google" will return plenty of reading material.

Likewise, each note in a chromatic scale, has a different degree of perceived "tension". The root note would be "home", where something is harmonically resolved.

Something like the root sharped would be an extreme amount of tension. Notes that aren't in the natural series of harmonic overtones (like a flat fifth) are perceived as unsettling. A fifth would have a feeling of leading back to the root.

If I was going to associate color with given notes, I would be thinking more along these lines than in terms of deriving color choices based on the light spectrum. Aspects like bpm, minor/major, if the song is in an exotic modality, if there are particular "shout sections" (eg., not shouting, but a jazz term for when every instrument emphasizes something at the same time)....these would all be things that would influence my own choice of relating color to various notes.

This isn't saying going down the road of investigating stuff like light spectrum and how it relates to sound is wrong, but only to relate what works for me and why, and what I think is probably a way of picking colors that resonates with people more universally.

dust's picture
Re: Audio Spectrogram

its interesting colors and music etc... i like to read about it but but in practice i'm a bit lazy and rather just pic a hue i like and go with it or set colors to random.

although if you actually think about and plan it all out there is some sort of cross modal relationships to color and sounds.

lets hypothetically say the notes A, C, and E map out to R, G, and B. when the notes ACE are played polyphonically there must be some overlap between the notes that would be a mixing of these colors. so i suppose theoretically regardless of what color's you use you could infer the in between colors by some percentage of how far apart the notes are.

for example if you set up your colors for lets say arbitrarily a chromatic A minor scale then you should be able to either infer the chromatic major scale by some sort of distance percentage to the root. the major chromatic scale could be the exact opposite colors of the minor scale i have no idea i'm making this all up.

the whole point is yes light and color clearly can be mapped but you can also map any color to any sound so when doing these color music type of scores one really needs to make a clear abstract formulation as to why and or what the colors are representing sonically and visually.

whether that is trying to map the sound and light spectrums together in some weird octave relationship or making up some other weird map of your own representative the feeling a particular key can invoke. otherwise then it is just a visualizer..

LuxAqu's picture
Re: Audio Spectrogram

Again thanks for the info. I might add some effects to it with an RII patch.

Though what is the Aurora developers example?

cybero's picture
Re: Audio Spectrogram

The Aurora Developer's example is a 10.6 example, can be found in Apple's sample code section of the Developer's Connection.

LuxAqu's picture
Re: Audio Spectrogram

Yeah I used this equation:

theta = 2pi log2( f / C )

(Thats theta equal 2 times pi times the log base 2 of f divided by C)

Where theta is the "octave angle" in radians, f is the frequency of the given note in Hz, and C is the frequency of the note in Hz chosen to be the root/origin.

The octave angle is the just the angular distance around a circle that represents the octave. Musically speaking each half step would be 1/12th of the circle

In my original theory the 12 basic notes of the chromatic scale and the 12 basic colors (3 primary, 3 secondary, 6 tertiary) correspond 1 to 1, but more importantly it's the harmonic intervals that I am interested in, e. g. major thirds, perfect fifths, diminished fifths.

Pictures here: http://lukenimtz.com/archives/806

What's really interesting is seeing the chords of color. You can see them in this visualizer as they shift with the music. You could even see exactly which chords they are if I labeled the circle with the 12 musical notes. In any case it think its really cool to watch. =]

In this visualizer however, I did not use any sort of 12th divisions, and the AudioTools frequency spectrum would work well with such divisions . I just used the octave angle formula to map the frequency of each band directly to its corresponding hue. In this case I set the note C to correspond to the color red. I did this for a number of reasons, though ultimately it's a matter of convention. In my view its not that certain notes objectively correspond to certain colors, but it's the harmonic intervals that colors and notes have in common that might correspond. I think that might be what gtoledo3 was getting at.

Then again most music is just about a certain vibe that conjures up images, places, and people... something visualizers cant do yet. ;]

gtoledo3's picture
Re: Audio Spectrogram

Yeah, that's "sort of" what I was getting at.

I mean that there is probably a stronger correlation between intervals of a given key and the emotional response they cause, and the emotional response that various colors cause, than there is between the emotional response a given color causes in comparison to some color association dictated by the absolute frequency of a note.

So, say C is red in "the land of light wavelengthy stuff", but people tend to associate red with anger or sexual passion. Then if you are using red as the color to represent "C", when making a visualizer for the Beatles song "Hello Goodbye" you have just made a woefully inappropriate choice based on applying a theory that isn't functional to produce the intended result.

I tend to reject the kind of light=note=color theory, because a note that's sharped or flat looks essentially like the color next to it. Where is the tension in that unless it's to the point of a color clash (doesn't happen in my experience)?

To me, this kind of theoretical light=note=color stuff winds up looking very "computer programmer art"-ish (eg., NOT a good thing).

usefuldesign.au's picture
Re: Audio Spectrogram

Lucky so much Blues got played in G historically or we'd have to go back and do the whole birth of the blues thing again ;)

LuxAqu's picture
Re: Audio Spectrogram

Its true that sometimes whole songs seem to conjure up one predominate color with their mood, not to mention textures and shapes... and people and places...

But red has way more personality than you give it credit for, and any given song is much more colorful that just one color using this scheme. But like I said, the choice to associate a particular note to a particular color is ultimately a matter of arbitrary convention.

The color intervals and music intervals have many characteristics in common. You give a perfect example; colors that are immediately adjacent in hue ARE dissonant. Color clash is exactly what happens.

And what's wrong with computer programmer art?? Those musical sorting algorithms by cybero are awesome!

My most synesthetic friends enjoy this visualizer, and they're the real masters of this sort of thing. =]

cybero's picture
Re: Audio Spectrogram

I really must give credit, where credit is due LuxAqu :-)

In the case of the sorting algorithms, the video link posted here is actually by Aldo Cortesi on sortvis.org.

& yes, they are awesome [just not done by me in that particular post].

Now some of my work has begun to be more generative, which began with my Sound Square composition.

Synaesthesia might well suit / enable creative and original visualizations .

cybero's picture
Re: Audio Spectrogram

Perhaps you ought to try the absolutely free Capture It - it will capture in .mp4 format and won't watermark your copy.

It doesn't record the audio, but you can easily resync that with your recorded video in GarageBand [Movie track template] and export as a .h264.

Main caveat with this is , as I've recently found out when trialling the application is that it only captures 60 seconds of video. Could be used to get a song minute by minute. Ah well.

LuxAqu's picture
Re: Audio Spectrogram

Oops, yeah the sorting algorithms you posted I should say...

What's your Sound Square composition?

cybero's picture
Re: Audio Spectrogram

Sound Square - the links on that post, not in the Composition Repository.

LuxAqu's picture
Re: Audio Spectrogram

I got a bunch of errors when I tried to open it, but I really like the video with the line and the dot dancing around.

So is it sort of "playing" the images that are placed in the grid?

I've seen that tone matrix site before. Very cool. Have you seen the stuff on Wolfram Tones? http://tones.wolfram.com/

cybero's picture
Re: Audio Spectrogram

Doh, I forgot you are on Leopard.

That Sound Square composition needs 10.6 as it uses the 3D Audio Player patch [OpenAL] - no wonder you got a shed load of errors !

Yeah, I'd heard of the Wolfram tones before, a nice way to doodle out some nicely calculated midi files. Neat interface too.

dust's picture
Re: Audio Spectrogram

people need to give sound and light more credit in regards to their cross modal relationships, phenomenologically sonoluminescence proves there is connection between sound heat and light. now if you combine sonoluminescence with the pythagorean book "Music of Shperes" and theoretically some interesting things happen.

example given an element has a key note and knowing that H20 is a compound of these elements then playing those elements in chord should break the compound into its integral elements.

meaning if you recorded the key note of hydrogen and oxygen separately then play them together you could split water with sound. like i said its all theory and i am crazy but its pythagorean so arguing with me about it is pointless either way things are connected together in ways we have yet to comprehend.

dust's picture
Re: Audio Spectrogram

so i wasn't to far off with what i was saying in comparison to your theory. im thinking if you apply your formula to what i was saying the exact opposite of the minor chromatic wouldn't be the be the major chromatic but harmonically would be 3 semitones up or down from the diatonic root ? is that right ? hmmm its hard to do that formula in my head. is there a quartz file to this project, i liked your metaphysica by by the way...

LuxAqu's picture
Re: Audio Spectrogram

How should I go about sharing the quartz file? Do I just post it for download?

Yeah I am going to start working on a Metaphysical Index 2.0 that included this harmonic Music/Color theory, the I-Ching, the Tarot, Kabbalah and maybe even Elements of Matter... I might have to look into that sonoluminescence... anyways Blender 2.5 is awesome.

cybero's picture
Re: Audio Spectrogram

You can post a link from your own web site, use Box or similar as a download facility on that site or just use a plain URL.

Alternatively, you could post your finished composition to the Composition Repository on this site.

If your file includes assets that are loaded externally I think it's a good idea to make that comp run from a project folder, ensure file paths required work locally , archive the folder [.zip] and then post that project.

Often as not QC comps are totally self contained and thus can be uploaded exactly as they are - .qtz files

LuxAqu's picture
Re: Audio Spectrogram

dust's picture
Re: Audio Spectrogram

i have spent some time in my life, doing metaphysical art and what not... i have some pieces laying around in frames at my house. things like golden ratio's, tree's of life, eggs, thoth all combined into geometric patterns and stuff. its always interesting when someone gets what they are like my jewish friend noticed a sephirot in one of my frames. i have not done much of the metaphysical design type things since 2002 maybe ? when i was doing some work with the kaballah center.

ever since i went crazy in 2005 i have not done any metaphysical art because some people think its all fringe stuff and me being on the fringe all ready just invites lunacy. plus there is a whole bunch of miss informed pseudo meta-physicist's type out there polluting the internet with conspiracies that really have little knowledge of any esoteric, theosophical, hermetic qabalah, and or rosicrucian philosophies.

personally i consider myself hermetic or thats what i write when i fill my religious preference in a form. no body knows what that means, some people think it is another word for hasidic... the caduceus is part of my family crest, so part of understanding myself was to understand the symbology present in my own name etc....

so yeah thanks for the qc file i find things like this fascinating. i found an interesting graph the other day that can be related to the iChing and mckennas timewave zero...

http://en.wikipedia.org/wiki/File:Timewave_9_11_2001.png

as singularity approaches i have found a few other graphs that seem to end at the same time like hard drive capacity

http://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_time.svg

http://www.ncad.net/Co/mooreslaw_chart.gif

these are more than likely coincidental but i have seen the logarithmic graph of the exponential growth rate of human intelligence and this seems to converge the same timewave zero.

like i said i don't put much thought into these types of things anymore. some things are best kept a secret.

ohh my pythagorean theory i mentioned above based on the music of spheres requires you send sound on a pulse of light, which is strange as sound isn't supposed to travel at the speed of light, but it is needed in order to record the key note of an element. i'm sure there is some other way like using an audio microscope or something ?