The system addresses three persistent challenges in AI music generation: low generation efficiency, a pronounced trade-off between modeling granularity and computational complexity, and incomplete multimodal conditional control. Existing approaches fall into three broad categories. Autoregressive models like MusicGen predict the next latent vector in sequence, which limits parallel acceleration and invites error accumulation over long-range dependencies. Waveform-domain diffusion models like DiffWave operate directly on raw audio samples, but computational complexity grows linearly with audio duration, making minute-long music difficult to scale, and sample-level granularity struggles to capture high-level semantic structures such as thematic repetition and harmonic progression. Latent-space diffusion models like AudioLDM 2 adopt a two-stage compress-then-generate strategy, yet discrete vector quantization constrains the continuous expressive capacity of music audio, potentially yielding less natural timbral transitions and dynamic variations.

The new system adopts a two-stage compress-then-generate architecture centered on a pre-trained EnCodec neural audio codec. The codec’s encoder maps raw time-domain waveforms into sequences of continuous latent vectors through multi-scale downsampling convolutions, compressing 48 kHz audio to a frame rate of 75 Hz. That means only 75 latent vectors correspond to each second of audio, achieving a compression ratio exceeding 600 times compared with the original 1.44 million sample points in a 30-second clip. A residual vector quantizer then quantizes these continuous vectors into discrete representations through a cascaded structure of multi-layer quantizers, each layer performing vector quantization on the quantization residual from the previous layer to achieve high quantization accuracy while preserving signal fidelity.

The key novelty lies in a continuous expansion mechanism that preserves and injects quantization residuals to map the discrete latent space onto a continuous manifold. When training the diffusion model, the system retains the residual after the last quantization layer, which contains fine-grained spectral and temporal information lost during quantization. A learnable linear projection first aligns the dimension of this residual with that of the quantized vector, and the two are then added element-wise to produce the final continuous latent variable. Crucially, the pre-trained EnCodec decoder always takes the original quantized vector as input during waveform reconstruction, not the expanded version, so the mechanism enhances the diffusion model’s representational capacity while remaining fully compatible with the pre-trained decoder, requiring no fine-tuning or modification of the latter.

For the noise prediction network, the researchers constructed a deeply coupled hybrid architecture combining U-Net and Transformer. The network processes noisy latent vectors through four downsampling stages and four upsampling stages, with an initial channel number of 64 and a maximum of 512 at the middle block. Each stage contains two residual convolutional layers followed by a Transformer layer, and the middle block contains two Transformer layers, giving a total of 10 Transformer layers, each with 16 attention heads and a hidden dimension of 1024. Residual convolutions extract local detail features, while Transformer layers establish global temporal dependencies, enabling effective modeling of long-range musical structures. The middle block, where temporal resolution is lowest at approximately one-sixteenth of the original frame rate and the receptive field is largest, models the overall structure of the music at the most abstract semantic level. Time step embeddings are mapped through sinusoidal positional encoding and a three-layer multilayer perceptron, then injected into each residual convolutional layer via addition.

In terms of conditional control, the system supports three types of conditions: text descriptions, melodic reference audio, and rhythmic trajectories. Text is encoded by a pre-trained CLAP model, a contrastive language-audio pre-training model whose text encoder maps natural language descriptions into semantically rich 768-dimensional embedding vectors. Melodic references are processed through EnCodec itself, with a learnable modality embedding added to allow the model to distinguish between the reference melody and the generation target. Rhythmic trajectories input in MIDI format are represented as binary pulse sequences and encoded through a temporal convolutional network. When multiple conditions are provided simultaneously, their embeddings are concatenated along the sequence dimension, and classifier-free guidance allows users to independently adjust the control weights of text, melody, and rhythm. Setting the guidance strength to zero degenerates to unconditional generation, one performs standard conditional generation, and values greater than one make results more strictly follow user input.

To resolve conflicts between conditions, such as a text description calling for a lively dance track while a melody reference is a melancholic adagio, the researchers designed a hierarchical control strategy that categorizes conditions into global and local types. Global conditions like text descriptions are applied to all time steps through cross-attention, influencing overall style, mood, and structure. Local conditions like melody references and rhythmic trajectories are applied only to corresponding local regions through temporally aligned cross-attention. When conflicts arise, priority is given to local conditions, which provide more specific pitch and temporal information. This priority is indirectly implemented through the concatenation order of condition embeddings, with local condition embeddings placed in the latter part of the sequence and assigned distinct positional encodings. The system also supports a condition masking mechanism enabling users to draw control curves along the time axis, dynamically adjusting guidance strength over time steps through a user-defined time weighting function.

For inference efficiency, the system employs DDIM skip-step sampling, which parameterizes the reverse process as a non-Markovian process, enabling skip-step sampling without model retraining. The number of sampling steps can be reduced to one-twentieth of the original while keeping generation quality within an acceptable range. The researchers also introduced a progressive generation strategy built on a unified time-dependent attention mask that gradually expands the attention window from local to global as the denoising step decreases. In early sampling steps, the model primarily recovers low-frequency structures; in later steps, it progressively recovers high-frequency details. This mask is shared across all conditioning modalities and serves as a global scheduler for structural progression, while the independent per-condition guidance strengths operate orthogonally to it, modulating how much influence each modality exerts on the denoising direction at each specific time step. The decoupled design preserves the simplicity of a single mask while enabling multidimensional temporal control.

Training was conducted on the MTG-Jamendo dataset, a large-scale music corpus released by the Music Technology Group at Universitat Pompeu Fabra comprising 55,000 core tracks spanning 195 style tags. The researchers applied a two-stage filtering process, first retaining only tracks lasting between 30 and 180 seconds, then excluding tracks with perceptible encoding artifacts or low signal-to-noise ratios based on heuristic thresholds of PESQ below 3.0 and SNR below 20 dB. From the filtered pool of 55,000 high-quality tracks, they randomly split the data into 45,000 training pairs, 5,000 validation pairs, and 5,000 test pairs, ensuring no artist overlaps across splits to prevent data leakage. The MusicCaps dataset, containing 5,500 ten-second music clips with natural language descriptions, was used exclusively as a test benchmark for subjective evaluation and cross-dataset generalization assessment. Model training on a single A100 GPU took approximately six days over 200 epochs, with 1000 diffusion steps during training and 50 DDIM skip-steps during inference.

Experimental results demonstrate comprehensive superiority. The system achieves a Fréchet Audio Distance of 2.31, substantially lower than MusicGen’s 4.21 and DiffWave’s 6.75, and a PESQ of 3.71, significantly outperforming AudioLDM 2’s 3.42 and MusicGen’s 3.15. In subjective listening tests, 20 music professionals rated 100 randomly selected generated samples in a soundproof room under a double-blind design. The system earned a median Audio Quality score of 4.59 out of 5, with an interquartile range of only 0.09, and a median Naturalness score of 4.50, corresponding to relative improvements of 10.3 percent over AudioLDM 2, 6.6 percent over MusicGen, and 29.3 percent over DiffWave. A paired t-test between the system and AudioLDM 2 on Audio Quality scores yielded a p-value below 0.01, confirming statistical significance. In controllability experiments, increasing guidance strength from 1 to 4 raised the relative CLAP matching rate monotonically from 0.81 to 0.90 while naturalness declined only slightly from 4.50 to 4.45, with the optimal balance at a strength of 2.5 yielding an 8.6 percent improvement in CLAP alignment with only a 2.2 percent loss in naturalness. Heatmap analysis of joint text and melody guidance strengths showed the highest composite score of 0.85 at text strength 2.3 and melody strength 1.8, with the high-score region covering a broad parameter range, demonstrating stable output insensitive to the choice of guidance strengths. Ablation variants removing classifier-free guidance and the progressive generation strategy each degraded performance in their corresponding dimensions, indicating that CFG primarily affects condition fidelity while the progressive strategy mainly influences structural coherence. The researchers note that future work will focus on music-knowledge-guided conditional control and generalization validation in real-world scenarios.

Subject of Research: Artificial Intelligence

Article Title: Design and implementation of an end to end music audio generation system based on diffusion probabilistic model

Article References: Yang, Y., Luo, R., & Zhang, R. (2026). Design and implementation of an end to end music audio generation system based on diffusion probabilistic model. Discover Artificial Intelligence, 6(1), Article 1210. https://doi.org/10.1007/s44163-026-02280-2

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02280-2

Keywords: diffusion model, music generation, AI, neural audio codec, latent diffusion, EnCodec, generative AI, machine learning, Design, implementation, music, audio