Custom Track Markups & Quantization
This was the first music system I thoroughly proofed out and prototyped. The request was made to have things perform an action on beat to the music. The objective was to make it as easy as possible for the user to interact with this system right out of the box with little setup. Designers and audio could work together to curate a music track that was interactive and compelling through this system.
There were 2 separate key methods to interacting with the game's Quantization system. At its core, this system takes advantage of FMOD's callback functionality as well as Unreal's latent action manager. To use the Custom Markups system, on the Blueprint side of things we have a struct that contains information that allows the designer to listen for different parts of the song. Position, Marker Type, and Custom Marker. This uses FMOD's built in callback system to look for custom designated markers and was intended to be a lot more bespoke and explicit with where and when designers wanted certain actions to happen.
Position being the current position of the playhead in milliseconds. This would allow an action to be performed at an extremely precise point in any song since this is constantly tracked and listened for.
Marker Type was a defined enum containing basic musical structures that the node could listen for. This was great for basic repetitive actions that needed to be automated, such as lights flashing on the kick drum or a VFX playing upon a drum fill.
Custom Marker allowed a unique marker to be listened for based on a predefined string. For example if they wanted an explosion to happen during the climax of the song, we could set a marker in FMOD labeled "Explosion", and the designer could look for that label in Blueprints and tie any action they wanted when the marker was hit.
In this example below, Kick, Snare, Cymbal, Fill, and Fill 2 are all hooked up to unique actions. The words are simply playing animations on each marker manually defined in the FMOD track.
This is what a marked up track looks like. While normally it would be a pain to manually place all these, I used FMOD's extended scripting functionality to utilize some simple batch marker scripts making the process extremely simple. Markers placed in this manner can be edited at anytime to match whatever the designer is looking for Unreal.
Unlike custom markups in FMOD, the core quantization system is designed to be a lot more generic and overarching. The game had many quantized elements, so a central place they could all hook into was needed. The designer has less control over explicitly when things happen during a song, but the trade off is that the system is very performant, requires less upfront setup, and utilizes Unreal's latent action manager and leverages a separate thread purely for managing these actions. The vast majority of the core quantization system had to be written in C++ between our team's core programmers and myself.
Much like previous audio systems, a requirement was ease of use for the designers. Simple understanding of the types of note durations (quarter note, whole note, half note etc.) was all that was needed and the system handled the rest. FMOD just needed a tempo marker and quantization backend would automatically adjust for differences in tempo and time signature meaning it would correct itself if the music ever changed during gameplay or outright stopped.
