Procedural Music in AAA: Rise of the Tomb Raider and the Dynamic Percussion System
The use of procedural music in video games presents advantages and disadvantages. On one side, it allows better responsivity to game events (Phillips, 2014, p. 206), greater variation (Collins, 2008, p. 151; Stevens et al. 2016, p.132), and a more suitable cost (Phillips, 2014, p. 211).
On the other, the common problems of this choice are a less convincing sound palette (Jolly et al., 2008, min. 4.30; Sweet, 2015, p. 205), greater CPU usage, music that could be less interesting (Langston, 1986) and difficult implementation for a non-programmer. (Stevens et al., 2016, p. 132)
With the release of their Dynamic Percussion System (Intelligent Music System, 2015), featured and developed on Rise of the Tomb Raider (Crystal Dynamics, 2015), Intelligent Music Systems has provided a solution to create percussion scores procedurally which seems to solve these problems.
The system
While working on Rise of The Tomb Raider, the audio team was looking for a way to have music that could support successfully the complex combat system of the game.
Thanks to the big variety of AI and player states, the possible scenarios when an enemy is encountered are many: Lara (the player character) could be hidden while her target is either aware or unaware of her, then she could attack and kill him in a more quiet or exposed manner, and so on.
The states are many, and they change quickly: for these reasons, neither crossfading between different layers of music or resequencing seemed good options: too much was happening in little time. The team decided then to work on procedural music, with two main goals: it had to sound good and to support the combat system. To achieve the aim, crystal dynamics collaborated with intelligent music system in creating a middleware solution that could offer real-time music. (Lampersky et al., 2016)
How it works
The Dynamic Percussion System analyzes an existing percussion pattern to compose procedurally new music. This is done in the software’s designer tool (the Scheme Designer), which is where the assets are created.
First, the midi tracks of each (percussive) instrument of the original composition are fed into the software, and linked to their relative sample banks, to be analyzed.
Then, the user must create layers. Layers are used to group tracks (but they can consist only of one) that work together, like shakers or low, middle and high percussions.
Each layer is the procedural representation of the tracks linked to it, and for each one the software will then provide different options, like Volume, Pan, Solo, Mute, and, especially, an Energy value, which is the core of the system.
If the energy is at 100% the part will be like the one composed, but the system will play little variations to it on every loop to avoid repetition. If the energy value is moved down instead, the system will take out notes from the pattern but with musical intelligence, to keep the sense of the groove (Intelligent Music System, 2016).
In this way, quieter or more intense arrangements of the same pattern are created in real time, always with some difference. Bobby Tahouri, the composer of Rise of Tomb Raider, after hearing the system at work described it as “an ensemble of highly trained and sensitive performers who are completely devoted to your music, and who are improvising on it while the game is being played” (Intelligent Music System, n.d.).
Finally, the given values can be stored as presets (schemes) between which the system can switch immediately or after a specified time. In this way, the composition changes dynamically when the schemes are called by game events, in a fast and natural way made on a note level.
In Practice
The system and its fast adaptivity can be appreciated in the video below.
The percussions switch fast from different states, and this can be appreciated especially from minute 1.10. First, as the distance from the enemy decreases, the intensity of the pattern increases. Then, when the player starts aiming at the guard, the music grows again (1.17), to calm down when he is killed (1.20). To be noticed, also, the tight synchronization that occurs between the stingers and the percussion pattern when an enemy is killed. This can be achieved by using gates synced with the music beat (Stevens et al., 2016, p.394) or with functions like Custom Cues in Wwise.
As Lara approaches the camp and hides, the music increases again, becoming more intense when the enemies start looking for her (1.45), and even more when she comes out and is spotted. When the last guard is killed, the intensity is dropped down and the percussion pattern stops.
By looking at the video, it can be noticed how this system addresses the common advantages of dynamic music but also how it challenges the problems usually associated to it.
Sample Quality: although the memory limit for the samples was 30 megabytes (Lampersky et al., 2016), and that the definition of “good” is subjective, I would personally say that the overall quality is actually good and that the system has also a decent dynamic range. To stay in the memory limit, the samples were exported to mono and the tail was cut and supplied with the engine’s reverb.
Interest: As mentioned above, Bobby Tahouri was positively surprised by how the system interprets the provided patterns. By using pre-composed music and because it relies on snapshots (the schemes) created by the user, the interest of the music depends both on the algorithm itself and on the quality of the percussion arrangement provided. Moreover, the ability to create the right schemes and link them properly to the game events affects the final quality too. Data driven algorithms seem the best way to handle procedural music or sound design: a similar approach was used in fact in EA’s Spore (Jolly et. al, 2008) and in Hello Games’ No Man’s Sky with Vocalien (Sonar+D, 2016).
CPU: The software’s web page claims that the dynamic music system has “negligible CPU footprint” (Intelligent Music Systems, n.d.). At the moment of writing there is no specific data about the amount of memory needed, although the fact that the system was developed and implemented in an AAA title is promising.
Complexity of use: the provided Scheme Designer is a simple and efficient interface to create schemes, and it is fully explained in a series of tutorials hosted on the Intelligent Music System’s website. The system then can be implemented both in Wwise and FMod, and both Unreal and Unity plug-ins are under development (Intelligent Music Systems, n.d.).
Finally, the percussions are synced to the rest of the music, an example can be found at minute 16 of the video below. Thanks to the work of the audio programmers (“there is a lot of human interaction to make it sound like that” Lampersky, 2016, 10.52), the percussions support the music successfully, without feeling detached from the other layers.
Conclusion
The Dynamic Percussion System is a successful example of a hybrid approach to generative music in an AAA title, in which real time music supports and his supported by rendered tracks. Percussions were chosen because they allowed to reach the most fidelity with the lowest memory cost, but without this limit, one day all stems could be procedurally generated (Lampersky et al., 2016, min. 29.30). In this way, music could be influenced not only by game events, but also from itself, with more complex interactions between the layers, like the creation of counter melodies, call and responses or dynamic chords rearrangement .
Bibliography
Audiokinetic Inc. (2006) Wwise [Software]. Montreal (Quebec): Audiokinetic Inc. Available from: <https://www.audiokinetic.com/download> [Accessed 15th November 2016]
Bajakian, C. (2013) Audio Bootcamp: Producing Music for AAA Video Games [Online]. Available from: <http://www.gdcvault.com/play/1017670/Audio-Bootcamp-Producing-Music-for> [Accessed 15 November 2016].
Collins, K. (2008) Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design. Cambridge, Mass.; London: MIT Press.
Crystal Dynamics (2015) Rise of the Tomb Raider [multi-platform]. Tokyo: Square Enix
Intelligent Music Systems (2015) Dynamic Percussion System. N.d.: Intelligent Music Systems
Intelligent Music Systems (2016) TUTORIAL 3: Scheme Designer--Changing Parameters and Making Variations [Online video]. Available from: <https://www.youtube.com/watch?v=YCdTVxO97gw> [Accessed 15 November 2016].
INTELLIGENT MUSIC SYSTEMS (n.d.) [Online]. Available from: <http://www.intelligentmusicsystems.com/index.html> [Accessed 15 November 2016].
Jolly et. al (2008) Procedural Music in SPORE [Online]. Available from: <http://www.gdcvault.com/play/323/Procedural-Music-in> [Accessed 15 November 2016].
Langston, P. (1986) “(201) 644-2332 or Eedie and Eddie on the Wire: An Experiment in Music Generation. “ Morrisontown, N.J: Bell Communication Research.
Lampersky et al. (2016), Real-Time Procedural Percussion Scoring in ‘Tomb Raider’s’ Stealth Combat [Online]. Available from: <http://www.gdcvault.com/play/1023215/Real-time-Procedural-Percussion-Scoring> [Accessed 15 November 2016].
Phillips, W. (2014) A Composer’s Guide to Game Music. Cambridge, Massachusetts: The MIT Press.
Sónar+D (2016) Encouraging Chaos, the Use of Generative Sound in No Man’s Sky [Online video].
Available from: <https://www.youtube.com/watch?v=nUWFTLhZ1ro> [Accessed 15 November 2016].
Stevens et al. (2016) Game Audio Implementation: A Practical Guide to Using the Unreal Engine.
Sweet, M. (2015) Writing Interactive Music for Video Games: A Composer’s Guide. Upper Saddle River, NJ: Addison-Wesley.
