Challenges and Perspectives in Deep Generative Modeling

Images from current state-of-the-art generative models have unarguably led to impressive results. However, this comes with potential threats such as the spread of misinformation through generated images or unauthorized style imitations of certain artists.

To tackle the first problem, I’ll first present works from our group and collaborators, which do successful deep fake detection in the frequency domain of images. In the second part, I’ll talk about current work in progress where we try to (invisibly) watermark images, such that a model fine tuned on these images picks up on the watermark. This would allow us to certify that works from a certain artist have in fact been used.

Neural topic models are used to quickly get an overview of the central themes in large text corpora. They are typically trained only on content words, whereas other words related to syntax or style of the text are removed in preprocessing. Rather than relying on manually curated and static stop word lists, we propose a data-driven way of dynamically identifying the style component in text data. The idea is to differentiate between long-range and short-range dependencies in text data. Following previous work in linguistics and cognitive science we posit that content words tend to have long-range dependencies whereas style or syntax words have short-range dependencies. Topic models are good at learning topics by disregarding sequential information, exclusively focusing on long-range dependencies. Language models however process text sequentially to predict the next word given the immediate short-range context. In this talk I present a method for combining both types of models to automatically distinguish syntactic and semantic words. Instead of simply removing the syntactic words, we can also group them and use them for the text analysis alongside the semantic words. Results show that this data-driven way of separating the words leads to higher topic quality and better feature selection on a range of datasets. This may lead to better ways of disentangling content and style in text data in the future, aiding controlled text generation for longer texts and overcoming the current bottleneck in text generation which is the size of input prompt. It was discussed how the topics in neural topic models can be seen as experts and the topic distribution of each document as a product of experts. This view point could help to develop topic models with a flexible number of topics where topics could be added as new experts. An interesting direction might be to look at hierarchical topic models as a collection of experts where more general experts are located at the top of the hierarchy and combinations of experts or more specific experts would be at the bottom of the hierarchy. Furthermore, topic models could be integrated with foundation models for language. To do this, the tokenization and input format for topic models need to be unified and a general mechanism for conditioning on and extracting topics during text generation needs to be developed.

This talk highlights two recent theoretical results about deep generative models.

The first part is based on the work of Laszkiewicz et al. (2022) and introduces an approach for normalizing flows that allows to model distributions with heavy- as well as light-tailed marginals. We prove that the marginal tailedness of an autoregressive flow can be controlled via the tailedness of the marginals of its base distribution (i.e. the distribution of the hidden variables). This theoretical insight leads us to a novel type of flows that are based on a three-step procedure: first, estimating the marginal tail indices, second, accordingly defining a set of heavy-tailed and a set of light-tailed base distribution, and third, training a normalizing flow with data-driven linear layers.

The second part is based on recent work Damm et al. (2023). Here we show that for standard (i.e., Gaussian) VAEs the ELBO converges to a value given by the sum of three entropies: the (negative) entropy of the prior distribution, the expected (negative) entropy of the observable distribution, and the average entropy of the variational distributions (the latter is already part of the ELBO). The result implies that the ELBO can for standard VAEs often be computed in closed-form at stationary points while the original ELBO requires numerical approximations of integrals.

Both works serve as illustrative examples of the importance of improving our theoretical understanding of deep generative models to gain robust, exact, and efficient generative models.

Variational autoencoders (VAEs) are performant deep generative models, based on principled Bayesian inference techniques. However, in practice, people often use them with isotropic Gaussian priors over the latents, which makes all the different data points independent from each other. This often does not match our true prior beliefs, especially when working with time series data. In this talk, I gave a brief overview of some recent approaches using Gaussian processes (GPs) as priors in VAEs, highlighting their tradeoffs and historical development. During the discussion, we discussed some newer follow-up work that uses Kalman filters for the GP inference in the latent space. We also discussed the choice of kernel and what the computational tradeoffs are between a full variational GP and a variational Gauss-Markov process.

Generating structured data has many important real-world applications such as generating molecular graphs for (de novo) drug discovery or genome sequences of bacteriophages to combat antibiotic resistant bacteria. The design space of potentially interesting structures in such applications is typically huge, it has, for instance, been estimated that there are more than ${10}^{30}$ different phages and even more small, drug-like molecules. This is in stark contrast to the amount of structures known to have desired properties in these domains which is often less than one hundred; in a lead discovery task for an antagonist of a particular integrin thought to play a key role in idiopathic pulmonary fibrosis $24$ compounds were known to bind well from previous biological assays. For such applications, we propose an efficient active learning algorithm for generative models with domain-specific similarity measures. Similarity measures defined by domain experts are often not positive semi-definite and thus cannot be utilised by Hilbert-space kernel methods. They instead require more general Krein-space kernel methods which admit efficient learning by adapting Nyström approximations. Our active learning algorithm adapts the distribution of generated structures using a learned conditional exponential family model and leads to diverse sets of novel structures.

In recent years, enormous progress has been made to gather as much information as possible about an individual patient. The continuous adoption and integration of electronic medical records, linkage of data sources, and the advent of new diagnostic and digital monitoring technologies have led to an overwhelming amount of heterogeneous and multimodal clinical data. The ultimate aim is to utilize all this vast information for a medical treatment tailored to an individual patient’s needs. To achieve this goal, we develop new generative machine learning techniques capable of dealing with the challenges arising in the medical application domain. The methods we develop cover for example multimodal data integration, transparent model development, or probabilistic clustering models.

Current learning algorithms for generative models generally assume that data points are sampled independently, an assumption that is frequently violated in practice. We propose a likelihood objective of normalizing flows incorporating dependencies between the data points, for which we derive a flexible and efficient learning algorithm suitable for different dependency structures. Respecting dependencies between observations can improve empirical results on synthetic and real-world data, leading to higher statistical power in a downstream application to genome-wide association studies. So far, we have focused on normalizing flows due to their explicit likelihood modeling, but we can extend similar modeling approaches to other generative models. We have also assumed that the dependency structure is at least partially known in advance – we work on relaxing that assumption and learning low-rank approximations of dependency structure from the data.

Introduction: Generative Adversarial Networks (GANs) have emerged as powerful tools for generating realistic images, with conditional and causal models enabling fine-grained control over latent factors. In the medical domain, data scarcity and the need to integrate information from diverse sources present challenges for existing generative models, often resulting in low-quality images ill-suited for medical applications. To address this issue, we propose the Causal Multi-Source StyleGAN (CMSSG), an algorithm that leverages prior knowledge over the data distribution in the form of causal graphs over image covariates and conditioning to integrate heterogeneous data sources with differing underlying distributions. CMSSG learns from multiple data sources with divergent causal structures in parallel, effectively synthesizing the learned distributions for data generation. We present a proof-of-concept experiment demonstrating CMSSG’s ability to generate hand-written digit images with varying morphological features. Additionally, we apply CMSSG to generate brain MR images with heterogeneous characteristics from the UK Biobank and the Alzheimer’s Disease Neuroimaging Initiative (ADNI) datasets, illustrating its capacity to capture brain anatomical variations. Our proposed algorithm offers a promising direction for unbiased data generation from disparate sources.

Methodology: Our approach consists of two components: the causal component and the multi-source component. The causal component learns causal relationships between clinical and demographic characteristics and brain MRI features, facilitating the synthesis of images with specific attributes. The multi-source component enables the generation of synthetic data from multiple datasets with distinct causal models, fostering the creation of more diverse data with various characteristics from different distributions.

CMSSG learns from multiple data sources with divergent causal structures in parallel, effectively synthesizing the learned distributions for data generation. To the best of our knowledge, our work is the first to address multi-source heterogeneity in GANs within a principled causal framework.

Experiments and Results: We validate our CMSSG method by generating hand-written digits with distinct morphological features. We then apply CMSSG to generate brain MRIs with specific clinical and demographic characteristics. We train our model to learn causal relationships from two datasets in parallel: the UK Biobank dataset, focusing on the relationships between demographic characteristics and MRIs, and the Alzheimer’s Disease Neuroimaging Initiative (ADNI) cohort, examining the relationships between clinical dementia features and MRIs. Using CMSSG, we generate synthetic brain MRIs with controlled age, sex, brain volumes, and cognitive function (normal or impaired).

Our results demonstrate that CMSSG can synthesize high-resolution brain MRIs while realistically manipulating causal structures within the images. Although the Frechet Inception Distance (FID) score from CMSSG does not outperform CausalStyleGANs with a single causal model, it provides a new opportunity to manipulate multi-source causal covariates.

Conclusion: In conclusion, the Causal Multi-Source StyleGAN (CMSSG) represents a novel approach to address the challenges of data scarcity and biased datasets in medical image generation. By leveraging causal graphs and conditioning, CMSSG integrates heterogeneous data sources with differing underlying distributions to generate high-quality, diverse medical images. Our experiments demonstrate the efficacy of CMSSG in generating hand-written digit images and brain MRIs with specific clinical and demographic characteristics.

Future work will involve collaboration with medical experts for comprehensive quality assessment and exploration of potential applications of synthetic medical images. Additionally, further experiments should be conducted to improve the quality of synthetic images from joint causal covariates and design appropriate metrics for evaluating the causality of GANs in respect to anatomical factors. This will ensure that the model learns the correct anatomical pattern with the aging process.

Self-supervised learning has emerged as a powerful paradigm for machine learning, especially for drawing insights from unlabeled data. The key idea is to introduce auxiliary prediction tasks and to train a deep model to solve these auxiliary tasks. If the tasks are designed well, the trained model will be useful for a number of purposes, such as anomaly detection, feature extraction, and forecasting. Unfortunately, most successful approaches for SSL rely on domain-specific indictive biases and are, therefore, limited to individual use cases. In this talk, I present advanced self-supervised learning losses that facilitate domain-general self-supervised learning beyond images and text. Exponential family embeddings, for example, generalize word embeddings to provide insight into a wide range of applications. They are a useful tool for studying zebrafish brains in neuroscience, studying shopping behavior in economics, or studying language evolution in computational social science. Similarly, neural transformation learning (NTL) is a new general-purpose tool for self-supervised anomaly detection. While related methods in computer vision typically require image transformations such as rotations, blurring, or flipping, NTL automatically learns the best transformations from the data and generalizes self-supervised AD to almost any data type.

Neural image and video compression models have proven superior performance in rate-distortion and rate-perception tradeoffs compared to their classical counterparts. However, while most research still focuses on improving rate-distortion tradeoffs, neural compression models are currently much too resource-inefficient to deploy in real-world environments. This talks seeks to review strategies to maintain the strong performance of neural compression methods while aiming to reduce their runtime efficiency by 1-2 orders of magnitude. To this end, we propose three modeling and algorithmic improvements: (1) introducing lightweight decoders, (2) improving encoding at training time using semi-amortized variational inference, and (3) establishing probabilistic circuits as new models for efficient entropy coding in lossy compression.

Background: The internet and the world’s IT systems could not exist in their current form without data compression. With video streaming dominating consumer internet traffic, every percent of gained performance improvement will have a large economic impact. Neural codecs are potentially also better suited for new data formats, such as light fields for AR/VR applications, lidar data, or multi-view video. These technologies’ fast evolution will demand rapid prototyping, making learnable codecs appealing. Neural codecs also lack the common block-coding visual artifacts and can be “supervised” to allocate more bits to certain features of interest. Neural codecs are more flexible than traditional codecs and can be optimized for superior perceptual quality or other custom metrics at much lower bitrates.

While the focus of the neural compression community has been largely on improving the tradeoff between bitrate and distortion (or perceptual quality), neural compression methods are currently 1-2 orders of magnitude slower than their classical counterparts. This makes their real-time deployment highly impractical, e.g., downloading and unpacking a large machine learning data set such as ImageNet or decoding video in real-time. By drawing on resource-efficient architectures, iterative inference, and new entropy coding schemes based on parsimonious models, we argue that the community should seek to maintain the strong performance of neural compression methods while increasing their runtime efficiency ideally by 1-2 orders of magnitude, removing one of the major obstacles from widespread deployment of neural codecs in the real world.

Many compression applications, such as video streaming on Youtube, impose strict runtime limitations upon decoding while allowing a much larger computational budget upon encoding. In “Improving Inference for Neural Image Compression” [Yang, Bamler, Mandt, NeurIPS 2020], we exploited this asymmetry by improving the encoding process using a larger computational budget while leaving the decoder untouched. To this end, we searched for an improved discrete latent representation at test time using an annealing scheme. This way, we obtained 15-20% rate savings without modifying the decoder. Our result suggests that iterative inference may achieve state-of-the-art compression performance with more lightweight decoders.

In contrast to most existing work focusing on architectural improvements in non-linear transform coding, we stress the importance of algorithmic advances that have broad applicability to existing methods, e.g., by exploiting the asymmetrical resource budgets for encoding and decoding via iterative inference and/or by developing new paradigms for entropy coding. Ways to improve the resource efficiency of neural codecs include drawing on lightweight decoders, iterative encoding at training time, and advances in parsimonious generative models. Our goals are to accelerate transform coding while also proposing new architectures for efficient entropy coding based on recent work on lossless compression with probabilistic circuits.

One of the most popular frameworks to extract meaningful information from a vast amount of unlabeled datasets is representation learning. The latter should encode valuable information for downstream tasks, such as classification, regression, and visualization. However, there are many circumstances where purely data-driven approaches lead to unsatisfactory results that are inconsistent with the domain knowledge. On the other hand, deep generative models can encode physical laws and constraints into their generative process to obtain preferred representations of data, enabling exploratory analysis of complex data types. In this talk, I explored several approaches to incorporate domain knowledge, in the form of constraints, probabilistic relations, and prior distributions, in VAEs for static and temporal data with a focus on clustering. First, I introduced two different gaussian mixture prior distributions used in VAEs to enforce a clustering structure in the latent space. The first one [1] employs a categorical prior distribution on the clusters, while the second one /citehypergeometric uses a differentiable hypergeometric distribution to overcome the i.i.d. assumption of the input data. I then introduced the inclusion of domain knowledge in the form of probabilistic relations and survival information to obtain representations with a clear clustering structure for time series and survival data, using a mixture of Weibull distributions [3]. I then focused on instance-level constraints to guide the learning process toward a preferred configuration using high-dimensional data, using a Conditional Gaussian Mixture Model [4]. Last but not least, I showed how the proposed techniques can. be applied to real-world medical applications with a focus on cardiology. In cardiovascular medicine, to correctly quantify cardiac function and diagnose dysfunction, expensive and time-consuming medical imaging methods are often required. This might lead to a lack of available diagnostic modalities and inadequate patient care. The use of machine learning models based on affordable and minimally invasive diagnostics, such as echocardiography, might serve as a valuable assistant tool to enhance health care for people with cardiovascular diseases. However, a lack of large labeled datasets in the medical field prevents the use of supervised deep learning techniques. Therefore, there is a need for informed representation learning algorithms to leverage prior information on unlabeled datasets of cardiac ultrasound videos. The learned representation can then be used to solve further downstream tasks, such as diagnosis, anomaly detection, and denoising [5], [6].

Predictive models often need to be evaluated in dynamic, uncertain computational conditions. For example, a model deployed to a mobile device should be useful, despite a lack of computational power. Moreover, the same model ideally should provide better performance on devices with richer computational resources. Hence, there is a need for “early-exit” architectures that allow computation to be halted early, before the model has run to completion. Current architectures provide little-to-no strong guarantees about how these intermediate predictions relate to the final prediction produced by the full model.

In this talk, I describe a construction that provide one such guarantee. Specifically, we guarantee that the probability of the mode under the full model monotonically increases in the intermediate solutions as more computation is done. This is achieved by a product-of-experts approach, as its predictive distribution takes the form of an intersection of the experts. We demonstrate that this architecture can be realized for both real-valued regression and multi-class classification.

Recently, deep generative modeling has become the most prominent paradigm for learning powerful representations of our (visual) world and for generating novel samples thereof. Consequently, this has already become the main building block for numerous algorithms and practical applications. This talk will contrast the most commonly used generative models to date with a particular focus on denoising diffusion probabilistic models, the core of the currently leading approaches to visual synthesis. Despite their enormous potential, these models come with their own specific limitations. We will then discuss a solution, latent diffusion models, a.k.a. “Stable Diffusion”, that significantly improves the efficiency of diffusion models. Now billions of training samples can be summarized in compact representations of just a few gigabytes so that the approach runs on consumer GPUs. We will then discuss recent extensions that cast an interesting perspective on future generative models. In particular, retrieval augmentation during inference promises to significantly reduce model sizes by having powerful likelihood models focus on the composition of a scene rather than learning the training data. We will then highlight key aspects in which generative modeling will change in the future.

Generative models have been making large leaps in both quantitative fidelity and qualitative appeal. One class of models that has been a driving force behind these leaps is diffusion-based generative models. Diffusion-based generative models, or diffusions models for short, work by first corrupting data towards a known, fixed stationary distribution and training a model to undo those corruptions, and thus providing a means to generate data by uncorrupting a sample from the stationary distribution. The choice of corruption or inference process affects both likelihoods and sample quality. For example, it has been shown that extending the inference diffusion with auxiliary variables, making them multivariate, leads to improved sample quality. However, deriving training algorithms for each new inference diffusion is onerous requiring manually deriving stationary distributions and transition kernels. To simplify the training, we provide a recipe for likelihood training of multivariate diffusion models. In the first step, we derive a lower bound on the likelihood. Next, we show how the terms in the lower bound can be automatically computed and show how to parametrize inference diffusions using results from Markov chain Monte Carlo to target a specific stationary distribution. We study several different inference diffusions and demonstrate how to learn and the value of learning inference diffusions.

What distinguishes a realistic image from an unrealistic image? Despite tremendous progress in our ability to generate realistic images, we still lack functions that can reliably detect artifacts in images. Such a function would be of great interest in a variety of applications such as outlier detection, perceptual quality evaluation, neural compression, or neural rendering. The field of algorithmic probability provides many insights on a closely related question; namely, when is data a plausible sample from a distribution P? However, these results are not widely known in the machine learning community. In this short presentation, I will discuss how Kolmogorov complexity can inspire “universal critics” – functions that are able to detect unrealistic images without being trained on corrupted data.

The goal of source compression is to map any outcome of a discrete random variable $x\sim p_d(x)$ in a finite symbol space $x\in S$ to its shortest possible binary representation. Given a tractable model probability mass function (PMF) $p(x)$ that approximates $p_d(x)$, entropy coders provide such an optimal mapping. As a result, the task of source compression is simplified to identifying a good model PMF for the data at hand.

Even though the setup as described is the most commonly used one, there are restrictions to it. Entropy coders can only process one dimensional variables and process them sequentially. Hence the structure of the entropy coder implies a sequential structure of the data. This is a problem when compressing sets instead of sequences. In the first part of the talk, I present an optimal codec for sets [1]. The problem we encounter for sets can be generalized for many other structural priors in data. In the second part of the talk I thus investigate the problem. We generalize rate distortion theory for structural data priors and develop a strategy to learn codecs for this data [2].

Causal representation learning seeks to extract high-level latent factors from low-level sensory data. Most existing methods rely on fitting deep generative model to observational data, leveraging structural assumptions (e.g., conditional independence) to identify the latent factors. However, interventional data is prevalent across applications. Can interventional data facilitate causal representation learning? We explore this question in this talk. The key observation is that interventional data often carries geometric signatures of the latent factors’ support (i.e. what values each latent can possibly take). For example, when the latent factors are causally connected, interventions can break the dependency between the intervened latents’ support and their ancestors’. Leveraging this fact, we prove that the latent causal factors can be identified up to permutation and scaling given data from perfect do interventions. Moreover, we can achieve block affine identification, namely the estimated latent factors are only entangled with a few other latents if we have access to data from imperfect interventions. These results highlight the unique power of interventional data in causal representation learning; they can enable provable identification of latent factors without any assumptions about their distributions or dependency structure.

Since out-of-distribution generalization is a generally ill-posed problem, various proxy targets (e.g., calibration, adversarial robustness, algorithmic corruptions, invariance across shifts) were studied across different research programs resulting in different recommendations. While sharing the same aspirational goal, these approaches have never been tested under the same experimental conditions on real data. In this paper, we take a unified view of previous work, highlighting message discrepancies that we address empirically, and providing recommendations on how to measure the robustness of a model and how to improve it. To this end, we collect 172 publicly available dataset pairs for training and out-of-distribution evaluation of accuracy, calibration error, adversarial attacks, environment invariance, and synthetic corruptions. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting. Our findings confirm that in- and out-of-distribution accuracies tend to increase jointly, but show that their relation is largely dataset-dependent, and in general more nuanced and more complex than posited by previous, smaller scale studies.

Motivated by the recent success stories of generative modeling with diffusion models, we reconsider the training objective of hierarchical variational autoencoders (VAEs). By keeping the so-called forward process (from data to latent representations) fixed and recurrent, diffusion models are able to efficiently scale up training to very deep probabilistic models with many layers of latents, leading to impressive generative modeling performances. However, it is not so clear how to use diffusion models for applications that make use of the forward process, such as representation learning or data reconstruction tasks, as the fixed forward diffusion process progressively reduces mutual information with the original data up to the point of almost no correlation. For these types of applications, VAEs with their learned inference models are often a more natural choice of model architecture. Motivated by the empirical observation that deep hierarchies of layers of latent variables are crucial to the performance of diffusion models, we reconsider the trade-off between reconstruction error (“distortion”) and information content of the latents (“rate”) in hierarchical VAEs, i.e., VAEs with more than one layer of latents.

We observe first that the general rate/distortion trade-off of beta-VAEs [Alemi et al., ICML 2018] can be refined by splitting up the rate term into a sum of contributions from each layer of latents. Importantly, however, this separation is only possible if the inference model proceeds in the same direction as the generative model, i.e., opposite to the direction of the otherwise analogous forward process in diffusion models. Splitting the rate into layer-wise contributions allows practitioners to control the information content of each layer individually by introducing individual layer-wise Lagrange multipliers (“beta hyperparameters”). We argue that this increased control is useful in practice by grouping application domains of (hierarchical) VAEs into three categories depending on whether they use (i) only the generative model (generative tasks), (ii) only the inference model (representation learning tasks), or (iii) both (reconstruction tasks). We show both by deriving theoretical performance bounds, as well as by large-scale empirical evaluations that the optimal distribution of rate between layers of latents is different for the three categories of applications, and we provide practical guidance for choosing reasonable values for the beta hyperparameters for each category.

The main open question that we will consider in the future is if the proposed hierarchical rate/distortion theory can be used to train VAEs where the size of the latent representation does not necessarily match between the inference and the generative process. Here, the highest-level latents could represent the length of the next lower-level latent representation. Such a variable-length information bottleneck would allow training VAEs with transformer architectures for text, e.g., building on the work by Henderson and Fehr (arXiv:2207.13529). Treating the length as a (higher-level) latent variable would allow training scenarios where the length of the reconstructed text does not necessarily match the length of the original text. The hope is that this would allow a VAE to more freely rephrase text, and that the individual beta-hyperparameters would allow controlling the length and the diversity of reconstructed text separately.