Why adversarial diffusion trains more stably than GANs: a local Jacobian view

1The question

Modern generative models differ sharply in training reliability and sampling speed. GANs sample in one step but are notoriously unstable and prone to mode collapse. Diffusion models achieve excellent mode coverage and train reliably thanks to a well-defined scalar objective, but their parametrisation needs many small denoising steps, which makes sampling slow.

A line of work reduces the number of steps by replacing the ELBO with an adversarial objective at each denoising step. Xiao et al. conjectured that with step-wise denoising, even purely adversarial training is stable. That conjecture poses a clean question:

Is the reliability of diffusion models primarily due to their KL-based scalar objective, or can the step-wise denoising structure itself improve the local optimisation geometry of adversarial training?

Previous work has analysed GAN training stability from theoretical and optimisation standpoints, but to the best of our knowledge this is the first analysis of how the step-wise denoising process changes the optimisation geometry of adversarial training. The contributions are: (1) a unified local-Jacobian comparison of GAN, adversarial diffusion and ELBO diffusion training; (2) a derivation, for adversarial diffusion, of the \(c_t\)-weighted timestep averaging of the coupling term that controls rotational dynamics; and (3) validation of these predictions with diagnostic plots (Jacobian spectra and the per-timestep coupling proxy \(c_t A_t S_t\)) and learning-rate sweeps on 2D mixtures.

2Training as a dynamical system

All three objectives are trained by gradient updates, so all three can be studied as discrete dynamical systems and compared through the Jacobian of the update map near an equilibrium.

The GAN objective, following Mescheder et al., is written with a smooth \(f\) satisfying \(f'(0) \ne 0\) and \(f''(0) < 0\):

\[ \min_\theta \max_\psi \; \E_{\pn(z)}\!\left[f\!\left(D_\psi(G_\theta(z))\right)\right] + \E_{\pd(x)}\!\left[f\!\left(-D_\psi(x)\right)\right]. \]

The adversarial games are trained by simultaneous gradient ascent, while the scalar ELBO is trained by plain gradient descent:

\[ F_h(\theta,\psi) = (\theta,\psi) + h\,v(\theta,\psi),\quad v = \bigl(-\nabla_\theta L,\; \nabla_\psi L\bigr) \qquad\text{resp.}\qquad F_h(\theta) = \theta - h\,\nabla_\theta \mathcal{L}. \]

A Nash equilibrium is a fixed point of \(F_h\), and by classical fixed-point theory it suffices for local convergence that the spectral radius of \(\nabla F_h\) at that point is below one. If an eigenvalue of \(\nabla v\) is \(a + ib\) with \(a < 0\), the corresponding update eigenvalue is \(\lambda = 1 + h(a + ib)\), and

\[ |\lambda| < 1 \iff 0 < h < \frac{-2a}{a^2 + b^2}. \]

This inequality is the whole story. The imaginary part \(b\) measures rotation of the dynamics around the equilibrium, and it enters the denominator: a nonzero \(b\) shrinks the window of admissible learning rates. The same rotation that makes min–max training oscillate is what makes it learning-rate sensitive, and eigenvalues close to the imaginary axis may require intractably small learning rates to converge at all. A real spectrum (\(b = 0\)) removes the constraint entirely.

So the comparison reduces to a single quantity: how much imaginary part does each objective's Jacobian carry?

3Three objectives, one Jacobian

3.1GANs: full coupling

Without regularisation, GAN training dynamics can be purely cyclic and non-convergent. Adding the \(R_1\) penalty \(R_1(\theta,\psi) = \tfrac{\gamma}{2}\,\E_{x\sim \pd}\|\nabla_x D_\psi(x)\|_2^2\) and assuming sufficient expressivity, the regularised Jacobian at the equilibrium takes a block form:

\[ \tilde v'(\theta^*,\psi^*) = \begin{pmatrix} 0 & -K_{DG}^{\top} \\[4pt] K_{DG} & K_{DD} - L_{DD} \end{pmatrix}, \qquad K_{DG} = f'(0)\,\E_{\pn(z)}\!\left[\nabla_{\psi,x} D_{\psi^*}(G_\theta(z))\,\nabla_\theta G_\theta(z)\right]\Big|_{\theta=\theta^*}. \]

Here \(K_{DD} - L_{DD}\) is symmetric negative definite on the normal space to the equilibrium manifold. For any matrix of the form \(\left(\begin{smallmatrix} 0 & -B^\top \\ B & -Q \end{smallmatrix}\right)\) with \(Q\) symmetric positive definite, the eigenvalues have negative real part and satisfy

\[ |\Imag(\lambda)| \;\le\; \sqrt{\lambda_{\max}(B^\top B)} \;=\; \|B\|_2 . \]

Applying this with \(B = K_{DG}\) gives the central bound: the rotational content of GAN training is controlled by the generator–discriminator mixed gradient \(\|K_{DG}\|_2\). The eigenvalues have negative real part, so training converges at small enough learning rates — but the imaginary parts are large and \(O(1)\), so that convergence is highly learning-rate sensitive.

3.2DDGAN: the same Jacobian, with the coupling averaged

Adversarial diffusion replaces the ELBO with a per-timestep conditional GAN objective on \(q(x_{t-1} \mid x_t)\), parametrising \(p_\theta(x_{t-1}\mid x_t) = \int \phi(z)\, q(x_{t-1}\mid x_t, x_0 = G_\theta(x_t,t,z))\,dz\) through the diffusion posterior:

\[ \min_\theta \max_\psi\; \E_{\pi(t) q(x_t)}\Bigl[ \E_{q(x_{t-1}\mid x_t)}\log D_\psi(x_{t-1}, x_t, t) + \E_{p_\theta(x_{t-1}\mid x_t)}\log\bigl(1 - D_\psi(x_{t-1}, x_t, t)\bigr) \Bigr]. \]

Grouping \(p_{\mathcal{D}}(x_{t-1}, x_t, t) := \pi(t) q(x_t) q(x_{t-1}\mid x_t)\) and \(p_\theta(x_{t-1},x_t,t) := \pi(t) q(x_t) p_\theta(x_{t-1}\mid x_t)\) turns this back into a GAN objective of exactly the form above — so DDGAN inherits the same block Jacobian. The only thing that changes is the coupling block \(K_{DG}\).

Pushing the DDGAN parametrisation through the chain rule and applying the diffusion-posterior identities of Ho et al., the mixed gradient decomposes as a \(c_t\)-weighted average over timesteps:

\[ K_{DG} = \frac{f'(0)}{T}\sum_t c_t\, K_{DG}^{t}, \qquad \|K_{DG}\|_2 \;\le\; \frac{|f'(0)|}{T}\sum_{t=1}^{T} c_t A_t S_t , \]

where \(K^t_{DG}\) is the per-timestep mixed gradient (equivalent to a single-step GAN Jacobian), \(A_t\) and \(S_t\) are the per-timestep discriminator and generator gradient norms, and the weights are the diffusion-posterior coefficients

\[ c_t = \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1 - \bar\alpha_t}, \qquad \alpha_t = 1 - \beta_t,\quad \bar\alpha_t = \prod_{i \le t}\alpha_i . \]

Two bounds on these coefficients do the work. First, \(c_t \le 1\) always, which follows from \(\bar\alpha_{t-1}\beta_t = \bar\alpha_{t-1} - \bar\alpha_t\) and monotonicity of \(\bar\alpha_t\). Second, under the VP-SDE schedule of Song et al., \(c_t \lesssim 1/t\), so \(\sum_t c_t \lesssim \log T\). Together:

\[ \|K_{DG}\|_2 = O\!\left(\frac{\log T}{T}\right) \quad\text{(assuming }\|K^t_{DG}\|_2\text{ does not grow with }t\text{).} \]

This is a strictly tighter bound on the imaginary parts than for one-step GANs, and it improves as the number of denoising steps grows.

3.3DDPM: no rotation at all

For ELBO-trained diffusion, the dynamics are ordinary gradient descent, \(F_h(\theta) = \theta - h\nabla_\theta \mathcal{L}(\theta)\), with Jacobian \(J = I - h\,\nabla^2_\theta \mathcal{L}(\theta)\). Since \(\nabla^2_\theta \mathcal{L}\) is the Hessian of a scalar objective it is symmetric, so \(J\) has a purely real spectrum:

\[ \Imag(\lambda) = 0 \quad \text{unconditionally}, \qquad \nabla^2_\theta \mathcal{L}(\theta^*) \succeq 0 \ \text{at a local minimiser.} \]

Gradient descent on the ELBO simply lacks the rotational components characteristic of min–max dynamics — and unlike the DDGAN result, this holds regardless of realisability.

GAN  (T = 1)

max |Im λ| ≈ 26

DDGAN  (T = 8)

max |Im λ| ≈ 2.7

DDPM  (ELBO)

max |Im λ| = 0

4Why this is not just a rescaled learning rate

The obvious objection to the \(1/T\) prefactor is that it looks like an effective change of step size, which a larger \(h\) would undo. It is not, and the reason is structural: the \(c_t\) weight enters only the off-diagonal coupling, through \(\partial \mu(x_t,x_0,t)/\partial x_0 = c_t I\). The discriminator block carries no such factor. Instantiating it with the DDGAN distributions gives

\[ K_{DD} - L_{DD} = \frac{1}{T}\sum_{t=1}^{T}\left(K^t_{DD} - L^t_{DD}\right), \]

an unweighted average of \(T\) per-timestep blocks, each \(O(1)\), hence \(O(1)\) overall. What shrinks is therefore the ratio of the rotational (off-diagonal) part of the Jacobian to its dissipative (diagonal) part — not the vector field as a whole. That change in conditioning cannot be reproduced by rescaling \(h\).

5Experiments

We test three predictions of the analysis: that ELBO diffusion is least learning-rate sensitive; that adversarial diffusion is less sensitive than one-step GANs; and that increasing denoising depth \(T\) reduces the rotational magnitude of the Jacobian.

The setting is deliberately lightweight, so that the Jacobian can be formed and diagonalised exactly rather than estimated: 2D toy mixtures — 8-Gaussians (modes on a circle) and 25-Gaussians (modes on a grid). All models share an MLP backbone with the VP-SDE schedule of Song et al. We report the off-mode rate, the fraction of samples falling outside \(3\sigma\) of every mode, and declare a run successful if all modes are covered and the off-mode rate is below 5%.

Qualitative behaviour

On 8-Gaussians, GAN and DDGAN both exhibit ring-like inter-mode mass, whereas DDPM concentrates near the mixture components. Over 5000 samples:

Adversarial objectives leak mass between modes; the scalar ELBO objective concentrates on them.

Depth kills rotation

Learning-rate robustness

The predicted ordering — DDPM \(>\) DDGAN \(>\) GAN in stability — is what the sweeps show, and it follows directly from how much imaginary part each objective's Jacobian carries.

6Conclusion

We compared GAN, adversarial diffusion and ELBO-trained diffusion through the local Jacobian of their training dynamics. Min–max objectives show rotational components, which explains learning-rate sensitivity even when training converges. In adversarial diffusion, timestep averaging and \(c_t\)-weighting reduce the effective generator–discriminator coupling, yielding smaller imaginary components than in one-step GANs. ELBO-trained diffusion is the most stable in our setting, consistent with its scalar objective yielding a real spectrum near optima.

Overall: adversarial diffusion can improve GAN-style stability via timestep averaging, but scalar objectives remain the cleanest route to eliminating rotational dynamics near optima.

7Scope and limitations

• The analysis is local: it describes convergence in a neighbourhood of a non-singular Nash equilibrium, and says nothing about the global path taken to get there.
• The \(O(\log T / T)\) bound assumes \(\|K^t_{DG}\|_2\) does not grow with \(t\), and the logarithmic bound on \(\sum_t c_t\) is specific to the VP-SDE schedule.
• Experiments are 2D mixtures, chosen so the Jacobian can be formed exactly. They test whether the predicted diagnostics move in the predicted direction; they are not evidence about image-scale training.

8References

1. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. NIPS, 2014.
2. J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
3. U-C. Jun, J. Ko, J. Kang. Generative adversarial diffusion. ICCV, 2025.
4. L. Mescheder, S. Nowozin, A. Geiger. The numerics of GANs. NIPS, 2017.
5. L. M. Mescheder, A. Geiger, S. Nowozin. Which training methods for GANs do actually converge? ICML, 2018.
6. V. Nagarajan, J. Z. Kolter. Gradient descent GAN optimization is locally stable. NIPS, 2017.
7. T. Salimans, J. Ho. Progressive distillation for fast sampling of diffusion models. ICLR, 2022.
8. A. Sauer, D. Lorenz, A. Blattmann, R. Rombach. Adversarial diffusion distillation. ECCV, 2024.
9. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ICML, 2015.
10. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
11. Z. Xiao, K. Kreis, A. Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. ICLR, 2022.
12. Y. Xu, M. Gong, S. Xie, W. Wei, M. Grundmann, K. Batmanghelich, T. Hou. Semi-implicit denoising diffusion models (SIDDMs). NeurIPS, 2023.

Full statements, the complete DDGAN derivation, the bounds on \(c_t\), and all hyperparameters are in the appendices of the camera-ready PDF.