Generating Discrete Data Using Diffusion with a Continuous Relaxation

Abstract: Generative modeling of discrete data such as text poses challenges that differ greatly from those encountered with continuous data. A possible approach involves representing the discrete data in a continuous latent space, making it compatible with powerful continuous generative architectures such as diffusion and flow matching models. While diffusion models have been applied to such continuous representations, the critical choice of the initial embedding method itself has been less explored and tested theoretically. This research shifts the focus from the design of generative model alone to an investigation of the process used for creating the embedding, treating the generative model itself as a black box. We test various methods both theoretically and empirically, using techniques based on the Gumbel-Argmax property as well as practical approximations using other distributions. We evaluate these embedding strategies by applying them in several generative frameworks, comparing convergence rates, computational efficiency, training stability and the likelihood of the generated data. We demonstrate that the embedding choice can influence a model’s success and offer a practical guide for designing more efficient and stable models for discrete data generation.