3D Spectrogram Visualizer

Chris Donahue (cjd947)

CS384G Fall 2013 Final Project



Introduction

My final project is a 3D spectrogram visualizer created using OpenGL, GLEW, GLUT for the 3D graphics and display, KissFFT for an implementation of the FFT algorithm and JUCE for audio playback/file IO. A spectrogram is a visual representation of the Fourier transform of a signal over time. Spectrograms are typically visualized in two dimensions where the x dimension represents time and the y dimension represents the frequencies between 0 Hz and the Nyquist. Color intensity is usually employed as a third dimension to represent intensity of spectral energy at each frequency bin over time. Instead of or in addition to color, one could use a third spacial dimension to represent intensity of spectral energy.

Motivation

Spectrograms can augment a music listening experience by visualizing a pictorial representation of the frequency information while the audio plays simultaneously. Such spectrogram "visualizers" typically scroll from right to left, where the right most edge represents the spectrum of the current audio signal and everything to the left is a spectral history. Some artists even embed audio in their songs that will look like recognizable pictures in a spectrogram visualizer. An example of this can be seen in the Aphex Twin track Equation from his Windowlicker EP.


In order to see this face, one must be using a spectrogram with a logarithmic frequency axis. 2D spectrograms are fairly common visualizers but 3D spectrograms are far less common. I wanted to implement a 3D spectogram using JUCE to use as an assistive tool for music creation. JUCE is a multi-platform C++ audio toolkit that among other things can be used to create plug-ins for digital audio workstations. Implementing a 3D spectrogram in JUCE would allow artists to visualize their songs in the frequency domain in 3D in real time in any music creation software. Additional motivation stems from a desire to gain more practical knowledge of using OpenGL and GLSL.

Development

I was initially using JUCE for the UI of this project. However, JUCE's OpenGL classes are not compatible with GLEW and I wanted to use GLEW to manage my VBO's and shader programs instead of JUCE's custom methods. A scrapped JUCE UI is displayed below.


I switched to GLUT and a command line interface to manage the graphics/options in order to expedite the creation of this project. The project ended up as a modified version of this tutorial on scientific programming in OpenGL. I used a program called Photosounder to create a debug audio file to use to test the correctness of my renders.

After establishing the correctness of my rendering system, I modified the keyboard scroll function from the OpenGL scientific programming tutorial to synchronize scrolling to the playing audio file.

Usage

Options for opengl_spectrogram program:

Option Flag Default Value Description
--help Displays help message
--fft_size 1024 Sets the size of the FFT window
--fft_overlap 0 Sets the number of sample overlap for sequential FFT windows
--fft_window_type rectangle Sets the window function for the FFT, can be "rectangle" or "hanning" currently
--display_compression 1.0 Compresses the display of the FFT. Sets a hard cieling on spectral energy at this value times the max energy at any bin in the frequency-time domain. Other values are linearly scaled towards this cieling.
--spectrum_display_percent 1.0 Changes the amount of frequency bins to display to this value times the number of FFT output bins. Useful for creating better visualizations in a linear scale. Will be less useful once logarithmic frequency axis functionality is implemented.
--vertex_shader_file_path REQUIRED Path to the GLSL file to use as the vertex shader
--fragment_shader_file_path REQUIRED Path to the GLSL file to use as the fragment shader
--audio_file_path REQUIRED Path to the audio file to play/visualize (currently must be a WAV file)

Summary of Accomplishments

Used OpenGL and other technologies to create a synchronized, scrolling 3D spectrogram to visualize an audio file. Created command line interface to change various visualization options. Learned loads about scientific computing using OpenGL, knowledge that will be extremely useful for my research efforts in the future.

Known Issues and Future Work

When the texture representing the spectral energy is scrolling, the spectral history (which should be static) seems to jump around and vary. I am not entirely sure what is causing this, but I am looking into it and it will be fixed in a later release. Additionally, I am looking in to support for logarithmic frequency axis and different input audio formats. Eventually I would like this software to be able to visualize FFT data of real-time audio input for use as a plug-in for digitial audio workstations. Finally, this software is currently only tested on Linux.

Artifact

Used TechSmith Jing to capture a short, (low FPS) video of my spectrogram visualizer playing the Aphex Twin track Pulsewidth.


Links and References

Source Code (github)
JUCE Home Page
Kiss FFT
OpenGL Scientific Programming Tutorial