Presto! Distilling steps and layers for accelerating music generation.

Presto! Distilling Steps and Layers for Accelerating Music Generation.

Abstract

Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via preserving hidden state variance. Finally, we combine our improved step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Furthermore, we find our combined distillation method can generate high-quality outputs with improved diversity accelerating our base model by 10-18x (32 second output in 230ms, 15x faster than the comparable SOTA model) --- the fastest TTM model to our knowledge.


Song Describer Prompt Examples (Cherry-picked)

Random Song Describer prompts. A batch of 4 was generated and a human selected the best one.

Text
Presto (ours)
Base DiT
Stable Audio Open

1093792-Intense pop song with a base of piano and synthesizers

1143013-This song starts with a ukulele and builds up with percussion using claps and an acoustic guitar that plays the same rhythm as the ukulele with melody played on a xylophone and has a very upbeat feel

1078393-Groovy funky song with an intimate touch, perfect for a small self contained dance

108303-Delayed synth blips and electronic drums combine on an ambient electronica track

1169705-Beautiful and evocative piece of classical piano music with a nostalgic touch

1166070-This track sound like a Christmas song with bells as the melody and strings as support

103892-bluesy guitar with a slow repetitive rythm in a smoky room in latin america

1126719-Alternative electronic song with experimental sounds and unsettling vibe

1093793-Electronic disco music about love that builds up quickly and has a dance beat to it

1163315-A relaxing jazz song, for focusing, that evokes sentimental feeling

1113971-Progressive electronic song with an intro of African percussions

1162034-Step onto a cruise and sip your drink in your Hawaiian shirt while listening to this track under the afternoon breeze in StTropez

1157362-Its a instrumental song, sounds epic and grand, like a score of a movie, has drums, keys, strings It is kind of mysterious, sad, epic spacey

1151409-A lullaby turned dark in its melody begins on a plucky kalimba of some kind; after an intro, a flute takes over the melody and a harp and other orchestral instruments back it up

1159370-An electronic chill out track featuring some instrument similar to santur

10575-A track with elements from eurodance and indie guitars

1162014-A pop-rock track very close to a proto-punk sound

1169706-This song gives a narrative of discovery or working something out I could imagine hearing it at the end of a video game working out my companion was the bad guy or solving a mystery that was not obvio

1162697-A ukulele-driven track with a turnaround harmonic progression Some melodics pads, muted electronic organ sounds and a slide guitar enrich the backing track with simple and enthusiastic melodies

1156053-Genre seems to be Texas country music and I feel hopeful and felt like I'm in a farm with waiting for the sunset with animals walking around

1155158-A dramatic and emotional solo piano track modern and classic at the same time

1060600-Melodic jazz piece starting off with a solo sax intro and continues with smooth electric guitar and synthesizer riffs

1143012-A joyful and sweet instrumental music track which is leaves the listener with positive emotions

1143152-Instrumental piano piece with a slightly classical touch and a nostalgic, bittersweet or blue mood

1169708-Western Classical music, piano, multiple sections, happy and nostalgic

1163322-This is a calm swing music with drums, synth and trumpet responding to each other and makes you want to bounce

1054342-A slow tempo guitar picking song of a folk genre which can be played on a road trip it evokes a relaxed sensation

1122392-Electronic music with a high energy vibe that can possibly be used for a spin class or dance floor

1135703-Contemporary trendy optimistic indie pop, with dirty drums, happy guitar comping and synthesizer solo

1119016-A garage rock-influenced song with lyrics about space which evokes psychedelia

Song Describer Prompt Examples (Random)

Random Song Describer prompts. One prompt outputs.

Text
Presto (ours)
Base DiT
Stable Audio Open

1055022-This song that features a piano, cello and oboe starts calmly and then arrives to a crescendo to give a triumphant atmosphere

108303-Delayed synth blips and electronic drums combine on an ambient electronica track

103887-Two electric guitars in conversation with each other, one with a wua-wua effect and the other with a strong delay effect

1051196-country-pop relaxing song with happy mood and acoustic guitar, ideal for young girls going out shopping or dreaming about their love

103892-bluesy guitar with a slow repetitive rythm in a smoky room in latin america

1061473-oh, this is kind of post punk music, remind me my high school memory

1009671-Upbeat fast tempo with a blues rock feel that one can dance

1063454-A deeply soothing track featuring two string instruments makes one peaceful with theirselves and the world

1036934-Indie alike track to time-travel to mid 2000s

1062831-Only instrumental and based on electronic samples that picks up as the song progresses

1066198-Feels like there is an argument happening between two people with the constant beat making it like time is progressing forward which can be used for either casual listening, reminiscing on a memory or

1050845-Synthetic orchestral piece in the style of a 90s war movie battle scene soundtrack, with prominent bass drums and dry brass section harmonies

1061369-Probably a three piece band with melodic guitar, heavy bass and drums playing an instrumental and energetic piece

1063332-Groovy instrumental funk rock track with occasional guitar solos that give you a feeling of longing

10575-A track with elements from eurodance and indie guitars

1063331-An instrumental hopefully positive track where the foreground melody features a classic guitar

1009672-A joyful and lively song that will make you want to dance right away

1054396-fast and fun beat-based indie pop to set a protagonist-gets-good-at-x movie montage to

1060600-Melodic jazz piece starting off with a solo sax intro and continues with smooth electric guitar and synthesizer riffs

1007274-acoustic guitar solo track with consistent rhythm and repeating progression, suitable for a relaxing afternoon tea

1004034-Electronic music that has a constant melody throughout with accompanying instruments used to supplement the melody which can be heard in possibly a casual setting

1053845-An electronic ambient track, with a dark tone The song progresses building the sections on top of each other A hollow percussion drum beat builds on top of a dark and soft harp, towards the end a viol

1051201-Im in a charismatic church and feeling sea sick from all the swaying to and fro

103333-This is an instrumental track based on electronic samples that can be used before the start of a movie to give a doomsday or post apocalypse feel

1066197-the song is moving piano is playing over long notes

1054342-A slow tempo guitar picking song of a folk genre which can be played on a road trip it evokes a relaxed sensation

1057890-A cinematic electronic soundtrack which announces the epic journey the protagonist will undertake in a post-apocalyptic world

1061384-An instrumental song that evokes some kind of a rush to go and declare your love

1033252-A 70s style british pop song with drums, guitars, a synth violin sounds and finally bright piano chords

1051193-A song that's intensely personal as its superficial lyrics that take you to nowhere