Echotron - WebGL Semantic & Extensible Music Visualization

Been working on a really fun experiment.

I call it the Echotron. Check it out here (WebGL capable browser required)

spiral and rings

I know what you’re thinking. “Another music visualizer, like there isn’t enough already!” Well, I like to think this ones a little different.

Semantic

Most music visualizers visualize the waveform of the music, analyzing it in realtime, and animating some state based on that feedback. That can produce some cool visuals, but I’ve always found the correlation between song events and and what the visuals do somewhat disconnected. Simply making the visuals pulse to the steady beat of the song is very tricky with simple realtime waveform analysis. It’s extremely difficult to make a machine perceive music in the same way that the human brain does.

And then there’s Echonest

Echonest has a very cool API feature. It will analyze the song and give you back a huge JSON file with incredibly detailed song metadata. Every beat, measure, section, and note is listed, including when it happens, how long it goes for, and even what tones occur.

Here’s an weensy sample of what they provide:

With this info we can make the visualization respond to semantic music events. When a beat happens, we can make the visualization pulse or jump. On an individual segment, we can highlight part of some object according the pitches within that note. Also (data not shown above) it provides data on measures/bars and sections (long stretches of similar style, like say a verse, or a chorus).

This is much closer to the way the human brain perceives the music. We can now make a visualization “dance” based on similar cues that allows a human to dance. The result, hopefully, is that the visual feels more connected to the actual music that is playing.

You can read more about the output from their analyzer here.

Extensible

The other part that makes this difference is extensibility. At the moment, there are not a lot of different layer types, but it’s designed to be a very simple engine that provides an interface that a WebGL doodler can quickly conform to and get something on screen. To make your doodle dance, you simply implement a method like onBeat and modify whatever state you need to change on each beat.

Each scene is a combination of three “layers”: Background, Midground and Foreground.

  • Background: Renders first, paints entire frame.
  • Midground: Renders second, typically textures or adorns background.
  • Foreground: Renders last, typically some sort of object animating front and center

Each “scene” is composed of a single random layer of each type. So adding more complexity and variation to the system is as easy as adding a new layer type.

But what’s a layer’s code look like?

Since the engine sets up the context, and music event callbacks for you, it takes very little code to do something simple. It’s based around THREE.js so managing positions, rotations, materials and other things is pretty straightforward.

The following example simply spins a cube around, and changes it’s color on every beat.

(exciting, eh?)

It’s easy to see just how simply you can elaborate on that. You can keep track of more complex state in the instance, update it every frame in update() and kick your numbers on event handlers like onBeat, onBar and onSegment.

Contributing

If you are a WebGL doodler or technical artist and want to contribute your pretties, checkout the repo on GitHub here

For now, you will clone the repo, add layers, and ask me to merge it back in. Though in the future I have plans to build a better system around contributions, allowing each layer to be in it’s own remote repo and pushed to a central server.

And Chrome Apps look like a super-cool use case for this as well. Stay tuned for that :)