Effective Interactive Music Systems
October 9, 2014, Audio Engineering Society Convention, Los Angeles—Winifred Phillips from Generations Productions talked about the nuts and bolts of interactive music in a session on game audio. The most important facet of game music is that it not like other music.
In the game “Frogger” the music starts with a moderate tempo, and accelerates as the frog moves across the stream. This change in tempo increases the intensity and is used in most games today. In “Assassins Creed” the game triggers change the music depending on the user’s actions and the situation.
These interactions of music and games requires the game and music share structures, with separate components and triggers to support the action. Interactive music requires audio files that are stacked in a vertical or horizontal structure and use some of the game data for activation and changes. In “Gods of War”, the music is part of the game navigation structure. The volume changes with proximity.
People have been interested in interacting with music since the mid-18th century when Hayden wrote some music that depended on the roll of some dice to determine measure sequences. This technique is now used in horizontal resequencing to change the order of the segments over time. “Speed Racer” used a number of component files with one set for a “zone” mode for a 15 second burst of hyper-speed. The main track changed to the zone mode and back again when the payer got to some trigger points. The music cross-fades between the two tracks as the digital markers identify the switching points.
“Tron 2.0” had 5 chunks of music, each 10-15 seconds long that covered 2-4 measures of music plus an ambient reverb tail. Advancing in the game changes the order of the music to make it less predictable and seemingly non-repetitive. This fabrication allows the gamer to spend a lot of time in the same area without getting bored by the same music. The action and a randomness generator juggles the chunk arrangements for freshness.
This type of sequential layering requires the developer to mix and master all of the components for smoother transitions. It is possible to use linear tracks in repeating patterns for the music. You an repeat the section until the gamer advances. One key is to make all of the chunks work well with most of the other chunks.
In vertical layering, the various channels carry separate tracks that are all relatively synchronized. The channels for fine control change the levels and the audio engine is a virtual mix engineer. Each component in a vertical layer has to be satisfying by itself. In “Little Big Planet 2”, there are 6 layers comprising drum, cute layer, a small women’s ensemble, the diva layer, and rock guitar and vocoder layer. There is contrast in the layers.
The approaches to take to use these techniques are additive, with a constant total volume, or interchange. An additive track will use complementary layers to indicate changes in the environment, and any layer can be dominant. In an interchange, not all layers can co-exist, so the layout is not as flexible. The lack of compatibility reduces the number of combinations possible. For example, in “Spore Hero” the music system can mix a comedy and exploration layer, but cannot use the comedy and epic layers together. An issue is the integration of the vertical layers with any existing linear music. This can be done in a custom mix as they do in cinema. The layers can change with the levels of the game.
It is possible to combine both horizontal and vertical layers in a game. The horizontal layers are separate measures of music while the vertical layers are multiple voices doing the same music. This effect is much like a MIDI player, with sound banks, instruments, and a player. The music data can be used to manipulate the music directly. An example is “Monkey Island” from Lucas Arts that changes instruments on the fly. MIDI is still used in games and there are tools for user generated content using MIDI sequences.
Another challenge in games is limited storage. “Shrek the 3rd” on the Nintendo DS has access to 1.5 MB total storage. within this space are background mood and individual character melodies. One way to get around this limitation is to have a sound bank of highly compressed chunks of files with integrated instruments. This technique is used in a lot of casual games and with a audio player can change tempo over time.
Finally, there are generative systems. These systems use indeterminacy and randomness to change the tracks. Some of the audio engines use probability to calculate which of up to 32 melody fragments to use based on some conditional statements. This unpredictable nature avoids repetition and fatigue as the variables can also change. Advances in the underlying technologies has enabled increasing complexity in game audio and music. The games are even using longer linear music sequences in large sections of the games.
