Redirecting to original paper in 30 seconds...
Click below to go immediately or wait for automatic redirect
📄 Abstract
Abstract: Masked diffusion models have recently emerged as a flexible framework for
discrete generative modeling. However, a key limitation of standard masked
diffusion is its inability to effectively capture dependencies among tokens
that are predicted concurrently, leading to degraded generation quality when
dependencies among tokens are important. To explicitly model dependencies among
tokens, we propose Variational Masked Diffusion (VMD), a framework that
introduces latent variables into the masked diffusion process. Through
controlled experiments on synthetic datasets, we demonstrate that VMD
successfully learns dependencies that conventional masked diffusion fails to
capture. We further validate the effectiveness of our approach on Sudoku
puzzles and text datasets, where learning of dependencies among tokens improves
global consistency. Across these domains, VMD enhances both generation quality
and dependency awareness, highlighting the value of integrating variational
inference into masked diffusion. Our code is available at:
https://riccizz.github.io/VMD.
Authors (3)
Yichi Zhang
Alex Schwing
Zhizhen Zhao
Submitted
October 27, 2025
Key Contributions
Introduces Variational Masked Diffusion (VMD), a framework that enhances masked diffusion models by incorporating latent variables. This allows VMD to explicitly model dependencies among concurrently predicted tokens, leading to improved generation quality and global consistency, particularly in domains like text and combinatorial puzzles.
Business Value
Enables the generation of more coherent and contextually relevant discrete data, useful for applications like creative writing, code generation, and solving complex combinatorial problems.