<div id="readability-page-1" class="page">




<header>
 
</header>


<div>
 
 <a href="/categories/generative">Generative</a><h2>[WIP] Diffusion, Flow Survey</h2>
 By YoungJoong Kim
 
 
 09 February 2025
 
 
 
 <img src="https://revsic.github.io/images/post/diffusion-survey/head.png" alt="[WIP] Diffusion, Flow Survey"><div>
 <ul>
<li>Survey of Diffusion, Flow Models</li>
<li>Keyword: Bayesian, VAE, Diffusion Models, Score Models, Schrodinger Bridge, Normalizing Flows, Rectified Flows, Neural ODE, Consistency Models</li>
</ul>
Abstract
2013년 VAE[<a href="https://arxiv.org/abs/1312.6114">Kingma &amp; Welling, 2013.</a>], 2014년 GAN[<a href="https://arxiv.org/abs/1406.2661">Goodfellow et al., 2014.</a>]을 지나 2020년의 DDPM[<a href="https://arxiv.org/abs/2006.11239">Ho et al., 2020.</a>]과 2022년의 Flow Matching[<a href="https://arxiv.org/abs/2210.02747">Lipman et al., 2022.</a>]까지, 생성 모델은 다양한 형태로 발전해 왔다. 기존까지의 생성 모델을 짚어보고, 앞으로의 방향성에 관하여 고민해 보고자 한다.
Introduction
Supervised Learning은 흔히 입력 데이터 $X$와 출력 데이터 $Y$의 데이터셋 $D$가 주어진다; $(x, y)\in D$. 이때 데이터셋 $D$의 분포 $\Pi(X, Y)$를 X와 Y의 Coupling이라 정의하자; $(x, y)\sim\Pi(X, Y)$ 
(simply assume the pdf $p_{X,Y}$ of $\Pi(X, Y)$ as $p_{X, Y}(x, y) = \delta_{(x, y)\in D}$ for dirac-delta $\delta$ and $(x, y)\in X\times Y$)
많은 경우에 Supervised Learning은 parametrized function $f_\theta: X \to Y$를 통해 $x\mapsto y$의 대응을 학습하고, 대개 조건부 분포의 likelihood를 maximizing 하는 방식으로 이뤄진다.
$$\hat\theta = \arg\max_\theta \sum_{(x, y)\sim\Pi(X, Y)} \log p_{Y|X}(f_\theta(x)|x)$$
만약 조건부 분포를 정규 분포로 가정한다면, 이는 흔히 알려진 Mean Squared Error; MSE의 형태로 정리된다.
$$\log p_{Y|X}(f_\theta(x)|x) \propto -||f_\theta(x) - y||^2 + C \implies \hat\theta = \arg\min_\theta \sum_{(x, y)\sim\Pi(X, Y)}||f_\theta(x) - y||^2$$
생성 모델(Generative Model)은 주어진 데이터의 확률 분포 학습을 목적으로 한다. 이는 probability mass function; pmf, 혹은 probability density function; pdf를 데이터로부터 추정하거나, 데이터 분포의 표본을 생성하는 Generator를 학습하는 방식으로 이뤄진다.
데이터 $X$의 분포를 $\pi_X$라 할 때, $\pi_X$의 pdf $p_X(x)$를 학습하거나, known distribution(e.g. $\mathcal N(0, I)$)의 표본 $z\sim Z$를 데이터 분포의 한 점 $x’\sim\pi_X$으로 대응하는 Generator $G: Z \to X$를 학습한다.
이 경우 대부분 사전 분포와 데이터 분포의 Coupling은 독립으로 가정하여(i.e. $\Pi(Z, X) = \pi_Z\times \pi_X$), parameterized generator $G_\theta$에 대해 log-likelihood를 maximizing 하거나; $\max_\theta \log p_X(G_\theta(\cdot))$, 분포 간 거리를 측정할 수 있는 differentiable objective $D$를 두어 최적화하기도 한다; $\min_\theta \sum_{(x, z)\sim\Pi(Z, X)} D(G_\theta(z), x)$.
전자의 상황에서 Generator가 $z\sim Z$의 조건부 분포를 표현하는 것은 자명하다; $G_\theta(z)\sim p_{\theta, X|Z}(\cdot|z)$. 우리는 $p_X$의 형태를 모를 때(혹은 가정하지 않을 때), 조건부 분포를 $Z$에 대해 marginalize 하여(i.e. $p_{\theta, X}$) 데이터셋 $X$에 대해 maximize 하는 선택을 할 수 있다; $\max_\theta \sum_{x\sim\pi_x}\log p_{\theta, X}(x)$
(후자는 GAN에 관한 논의로 이어지므로, 현재의 글에서는 다루지 않는다.)
조건부 분포를 marginalize 하기 위해서는 $p_{\theta,X}(x) = \int_Z p_Z(z)p_{\theta,X|Z}(x|z)dz$의 적분 과정이 필요한데, neural network로 표현된 $G_\theta$의 조건부 분포 $p_{\theta,X}$를 적분하는 것은 사실상 불가능하다(intractable).
만약 이를 $\Pi(X, Y)$에 대해 충분히 Random sampling 하여 Emperical average를 취하는 방식으로 근사한다면(i.e. Monte Carlo Estimation), 대형 데이터셋을 취급하는 현대의 문제 상황에서는 Resource Exhaustive 할 것이다. 특히나 Independent Coupling을 가정하고 있기에, Emperical Estimation의 분산이 커 학습에 어려움을 겪을 가능성이 높다. 분산을 줄이기 위해 표본을 늘린다면 컴퓨팅 리소스는 더욱더 많이 필요할 것이다.
현대의 생성 모델은 이러한 문제점을 다양한 관점에서 풀어 나간다. Invertible Generator를 두어 치환 적분(change-of-variables)의 형태로 적분 문제를 우회하기도 하고, 적분 없이 likelihood의 하한을 구해 maximizing lower bound의 형태로 근사하는 경우도 있다.
아래의 글에서는 2013년 VAE[<a href="https://arxiv.org/abs/1312.6114">Kingma &amp; Welling, 2013.</a>]부터 차례대로 각각의 생성 모델이 어떤 문제를 해결하고자 하였는지, 어떤 방식으로 해결하고자 하였는지 살펴보고자 한다. VAE[<a href="https://arxiv.org/abs/1312.6114">Kingma &amp; Welling, 2013.</a>, <a href="https://arxiv.org/abs/2007.03898">Vahdat &amp; Kautz, 2020.</a>]를 시작으로, Normalizing Flows[<a href="https://arxiv.org/abs/1505.05770">Rezende &amp; Mahamed, 2015.</a>, <a href="https://arxiv.org/abs/1807.03039">Kingma &amp; Dhariwal, 2018.</a>], Neural ODE[<a href="https://arxiv.org/abs/1806.07366">Chen et al., 2018</a>], Score Models[<a href="https://arxiv.org/abs/1907.05600">Song &amp; Ermon, 2019.</a>, <a href="https://arxiv.org/abs/2011.13456">Song et al., 2020.</a>], Diffusion Models[<a href="https://arxiv.org/abs/2006.11239">Ho et al., 2020.</a>, <a href="https://arxiv.org/abs/2010.02502">Song et al., 2020.</a>], Flow Matching[<a href="https://arxiv.org/abs/2209.03003">Liu et al., 2022.</a>, <a href="https://arxiv.org/abs/2210.02747">Lipman et al., 2022.</a>], Consistency Models[<a href="https://arxiv.org/abs/2303.01469,">Song et al., 2023.</a>, <a href="https://arxiv.org/abs/2410.11081">Lu &amp; Song, 2024.</a>], Schrodinger Bridge[<a href="https://arxiv.org/abs/2303.16852">Shi et al., 2023.</a>]에 관해 이야기 나눠본다.
VAE: Variational Autoencoder
<ul>
<li>VAE: Auto-Encoding Variational Bayes, Kingma &amp; Welling, 2013. [<a href="https://arxiv.org/abs/1312.6114">arXiv:1312.6114</a>]</li>
</ul>
2013년 Kingma와 Welling은 VAE를 발표한다.
References
<ul>
<li>VAE: Auto-Encoding Variational Bayes, Kingma &amp; Welling, 2013. [<a href="https://arxiv.org/abs/1312.6114">arXiv:1312.6114</a>]</li>
<li>GAN: Generative Adversarial Networks, Goodfellow et al., 2014. [<a href="https://arxiv.org/abs/1406.2661">arXiv:1406.2661</a>]</li>
<li>DDPM: Denoising Diffusion Probabilistic Models, Ho et al., 2020. [<a href="https://arxiv.org/abs/2006.11239">arXiv:2006.11239</a>]</li>
<li>Flow Matching for Generative Modeling, Lipman et al., 2022. [<a href="https://arxiv.org/abs/2210.02747">arXiv:2210.02747</a>]</li>
<li>NVAE: A Deep Hierarchical Variational Autoencoder, Vahdat &amp; Kautz, 2020. [<a href="https://arxiv.org/abs/2007.03898">arXiv:2007.03898</a>]</li>
<li>Variational Inference with Normalizing Flows , Rezende &amp; Mahamed, 2015. [<a href="https://arxiv.org/abs/1505.05770">arXiv:1505.05770</a>]</li>
<li>Glow: Generative Flow and Invertible 1x1 Convolutions, Kingma &amp; Dhariwal, 2018. [<a href="https://arxiv.org/abs/1807.03039">arXiv:1807.03039</a>]</li>
<li>NODE: Neural Ordinary Differential Equations, Chen et al., 2018. [<a href="https://arxiv.org/abs/1806.07366">arXiv:1806.07366</a>]</li>
<li>NCSN: Generative Modeling by Estimating Gradients of the Data Distribution, Song &amp; Ermon, 2019. [<a href="https://arxiv.org/abs/1907.05600">arXiv:1907.05600</a>]</li>
<li>Score-Based Generative Modeling through Stochastic Differential Equations, Song et al., 2020. [<a href="https://arxiv.org/abs/2011.13456">arXiv:2011.13456</a>]</li>
<li>DDPM: Denoising Diffusion Probabilistic Models, Ho et al., 2020. [<a href="https://arxiv.org/abs/2006.11239">arXiv:2006.11239</a>]</li>
<li>DDIM: Denoising Diffusion Implicit Models, Song et al., 2020. [<a href="https://arxiv.org/abs/2010.02502">arXiv:2010.02502</a>]</li>
<li>Rectified Flow: Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, Liu et al., 2022. [<a href="https://arxiv.org/abs/2209.03003">arXiv:2209.03003</a>]</li>
<li>Flow Matching for Generative Modeling, Lipman et al., 2022. [<a href="https://arxiv.org/abs/2210.02747">arXiv:2210.02747</a>]</li>
<li>Consistency Models, Song et al., 2023. [<a href="https://arxiv.org/abs/2303.01469">arXiv:2303.01469</a>]</li>
<li>Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models, Lu &amp; Song, 2024. [<a href="https://arxiv.org/abs/2410.11081">arXiv:2410.11081</a>]</li>
<li>DSBM: Diffusion Schrodinger Bridge Matching, Shi et al., 2023. [<a href="https://arxiv.org/abs/2303.16852">arXiv:2303.16852</a>]</li>
</ul>
<hr>
<details>
 <summary>TODO</summary>
 <ol start="0">
<li>Preliminaries</li>
</ol>
Oksendal SDE
<ul>
<li>Brownian Motion Model</li>
<li>Ito process</li>
<li>Ito Diffusion, Markovian Property</li>
</ul>
Normalizing Flows
<ul>
<li>Variational Inference with Normalizing Flows
, Rezende &amp; Mahamed, 2015. <a href="https://arxiv.org/abs/1505.05770">https://arxiv.org/abs/1505.05770</a>, <a href="https://revsic.github.io/blog/realnvp/">https://revsic.github.io/blog/realnvp/</a></li>
<li>Glow: Generative Flow and Invertible 1x1 Convolutions, Kingma &amp; Dhariwal, 2018. <a href="https://arxiv.org/abs/1807.03039">https://arxiv.org/abs/1807.03039</a>, <a href="https://revsic.github.io/blog/glowflowpp/">https://revsic.github.io/blog/glowflowpp/</a></li>
<li>Neural Spline Flows, Durkan et al., <a href="https://arxiv.org/abs/1906.04032">https://arxiv.org/abs/1906.04032</a></li>
<li>Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models, Huang et al., 2020. <a href="https://arxiv.org/abs/2002.07101">https://arxiv.org/abs/2002.07101</a>, <a href="https://revsic.github.io/blog/anfvf/">https://revsic.github.io/blog/anfvf/</a></li>
</ul>
Neural ODE
<ul>
<li>Invertible Residual Networks, Behrmann et al., 2018. <a href="https://arxiv.org/abs/1811.00995">https://arxiv.org/abs/1811.00995</a>, <a href="https://revsic.github.io/blog/resflow/">https://revsic.github.io/blog/resflow/</a></li>
<li>Neural Ordinary Differential Equations, Chen et al., 2018. <a href="https://arxiv.org/abs/1806.07366">https://arxiv.org/abs/1806.07366</a></li>
</ul>
<ol>
<li>Score model</li>
</ol>
<ul>
<li>Generative Modeling by Estimating Gradients of the Data Distribution, Song &amp; Ermon, <a href="https://arxiv.org/abs/1907.05600">https://arxiv.org/abs/1907.05600</a></li>
<li>Score-Based Generative Modeling through Stochastic Differential Equations, Song et al., <a href="https://arxiv.org/abs/2011.13456">https://arxiv.org/abs/2011.13456</a></li>
</ul>
<ol start="2">
<li>DDPM</li>
</ol>
<ul>
<li>Denoising Diffusion Probabilistic Models, Ho et al., 2020. <a href="https://arxiv.org/abs/2006.11239">https://arxiv.org/abs/2006.11239</a>, <a href="https://revsic.github.io/blog/diffusion/">https://revsic.github.io/blog/diffusion/</a></li>
<li>Diffusion Models Beat GANs on Image Synthesis, Dhariwal &amp; Nichol, 2021. <a href="https://arxiv.org/abs/2105.05233">https://arxiv.org/abs/2105.05233</a></li>
<li>Variational Diffusion Models, Kingma et al., 2021. <a href="https://arxiv.org/abs/2107.00630">https://arxiv.org/abs/2107.00630</a>, <a href="https://revsic.github.io/blog/vdm/">https://revsic.github.io/blog/vdm/</a></li>
<li>Denoising Diffusion Implicit Models, Song et al., 2020. <a href="https://arxiv.org/abs/2010.02502">https://arxiv.org/abs/2010.02502</a></li>
<li>Classifier-Free Diffusion Guidance, Ho &amp; Salimans, 2022. <a href="https://arxiv.org/abs/2207.12598">https://arxiv.org/abs/2207.12598</a></li>
<li>[Blog] Essay: VAE as a 1-step Diffusion Model
, <a href="https://revsic.github.io/blog/1-step-diffusion/">https://revsic.github.io/blog/1-step-diffusion/</a></li>
</ul>
<ol start="3">
<li>SDE &amp; PF ODE</li>
</ol>
<ul>
<li>Score-Based Generative Modeling through Stochastic Differential Equations, Song et al., 2020. <a href="https://arxiv.org/abs/2011.13456">https://arxiv.org/abs/2011.13456</a></li>
</ul>
<ol start="4">
<li>Rectified Flow &amp; Flow Matching</li>
</ol>
<ul>
<li>Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, Liu et al., 2022. <a href="https://arxiv.org/abs/2209.03003">https://arxiv.org/abs/2209.03003</a></li>
<li>Flow Matching for Generative Modeling, Lipman et al., 2022. <a href="https://arxiv.org/abs/2210.02747">https://arxiv.org/abs/2210.02747</a></li>
<li>Simple ReFlow: Improved Techniques for Fast Flow Models, Kim et al., 2024. <a href="https://arxiv.org/abs/2410.07815s">https://arxiv.org/abs/2410.07815s</a></li>
<li>Improving the Training of Rectified Flows, Lee et al., 2024. <a href="https://arxiv.org/abs/2405.20320">https://arxiv.org/abs/2405.20320</a></li>
</ul>
<ol start="5">
<li>Consistency Models</li>
</ol>
<ul>
<li>Consistency Models, Song et al., 2023. <a href="https://arxiv.org/abs/2303.01469">https://arxiv.org/abs/2303.01469</a>, <a href="https://revsic.github.io/blog/cm/">https://revsic.github.io/blog/cm/</a></li>
<li>Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples, Vouitsis et al., 2024. <a href="https://arxiv.org/abs/2411.08954">https://arxiv.org/abs/2411.08954</a></li>
<li>Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models, Lu &amp; Song, 2024. <a href="https://arxiv.org/abs/2410.11081">https://arxiv.org/abs/2410.11081</a></li>
</ul>
<ol start="6">
<li>Bridge</li>
</ol>
<ul>
<li>Diffusion Schrodinger Bridge Matching, Shi et al., 2023. <a href="https://arxiv.org/abs/2303.16852">https://arxiv.org/abs/2303.16852</a></li>
</ul>
<ol start="7">
<li>Furthers
Unified view</li>
</ol>
<ul>
<li>SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows, Nielsen et al., 2020. <a href="https://arxiv.org/abs/2007.02731">https://arxiv.org/abs/2007.02731</a>, <a href="https://revsic.github.io/blog/survaeflow/">https://revsic.github.io/blog/survaeflow/</a></li>
<li>Simulation-Free Training of Neural ODEs on Paired Data, Kim et al., 2024. <a href="https://arxiv.org/abs/2410.22918">https://arxiv.org/abs/2410.22918</a></li>
<li>Simulation-Free Differential Dynamics through Neural Conservation Laws, Hua et al., ICLR 2025. <a href="https://openreview.net/forum?id=jIOBhZO1ax">https://openreview.net/forum?id=jIOBhZO1ax</a></li>
</ul>
Fewer-step approaches
<ul>
<li>Progressive Distillation for Fast Sampling of Diffusion Models, Salimans &amp; Ho, 2022. <a href="https://arxiv.org/abs/2202.00512">https://arxiv.org/abs/2202.00512</a></li>
<li>Tackling the Generative Learning Trilemma with Denoising Diffusion GANs, Xiao et al., 2021.</li>
<li>InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation, Liu et al., 2023. <a href="https://arxiv.org/abs/2309.06380">https://arxiv.org/abs/2309.06380</a></li>
<li>One Step Diffusion via Shortcut Models, Frans et al,. 2024. <a href="https://arxiv.org/abs/2410.12557">https://arxiv.org/abs/2410.12557</a></li>
</ul>
Velocity consistency
<ul>
<li>
Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow, Want et al., 2024. <a href="https://arxiv.org/abs/2410.07303">https://arxiv.org/abs/2410.07303</a>
</li>
<li>
Consistency Flow Matching: Defining Straight Flows with Velocity Consistency, Yang et al., 2024. <a href="https://arxiv.org/abs/2407.02398">https://arxiv.org/abs/2407.02398</a>
</li>
<li>
[Blog] Essay: Generative models, Mode coverage, <a href="https://revsic.github.io/blog/coverage/">https://revsic.github.io/blog/coverage/</a>
</li>
</ul>

</details>

 </div>

 
 
 </div>




























</div>

revsic | ML Developer

Survey of Diffusion, Flow Models Keyword: Bayesian, VAE, Diffusion Models, Score Models, Schrodinger Bridge, Normalizing Flows, Rectified Flows, Neural ODE, Consistency Models Abstract

[WIP] Diffusion, Flow Survey

댓글