Diffusion models have become a leading framework for generative modeling, but their theoretical formulations and computational properties remain fragmented. Continuous diffusion models are commonly described through stochastic differential equations and probability-flow dynamics, whereas discrete models rely on continuous-time Markov chains or discrete flow-matching formulations [1, 2, 3]. Although these approaches share the principle of progressively transforming a simple reference distribution into a complex data distribution, their connection is not yet fully understood. This thesis will investigate whether continuous and discrete diffusion models can be described within a common operator-based dynamical framework, using Markov generators, evolution equations, and path-space distributions. Such a formulation should clarify the roles of model parameterization, numerical discretization, and conditional-independence assumptions, particularly in discrete models, where factorized reverse transitions can fail to represent strong correlations between dimensions.
Building on this unified perspective, the thesis will develop principled methods for making diffusion models faster and more compact. Distillation will be formulated as the approximation of a multi-step generative operator by a restricted few-step or one-step student model, connecting progressive distillation and consistency-based methods to inverse optimization and operator approximation [4, 5]. In parallel, the project will study model compression—including structured pruning, sparsification, quantization, and low-rank representations—while accounting for the time-dependent role of the denoiser along the generative trajectory [6]. A particular focus will be placed on discrete diffusion, where effective few-step generation requires student models that preserve the dimensional correlations implicitly created by repeated reverse transitions [7]. The expected outcome is a general, theory-guided toolkit for jointly reducing the number of sampling steps and model complexity while preserving generation quality across continuous and discrete domains.
References
[1] Y. Song et al. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR, 2021.
[2] A. Campbell et al. A Continuous-Time Framework for Discrete Denoising Models. NeurIPS, 2022.
[3] I. Gat et al. Discrete Flow Matching. NeurIPS, 2024.
[4] T. Salimans and J. Ho. Progressive Distillation for Fast Sampling of Diffusion Models. ICLR, 2022.
[5] Y. Song et al. Consistency Models. ICML, 2023.
[6] G. Fang, X. Ma, and X. Wang. Structural Pruning for Diffusion Models. NeurIPS, 2023.
[7] S. Hayakawa et al. Distillation of Discrete Diffusion through Dimensional Correlations. ICML, 2025.
