Generative Models for 3D Vision

Inspired by the seminal “Fake It Til You Make It” paper published in 2021, people have successfully trained ML models on synthetic data for research and product alike, demonstrating that the infamous domain gap can be overcome, even though the generated data still falls short in visual quality. Research has produced models trained on synthetic data that not only achieve parity with models trained on real data, but by now even surpass models trained on real data due to the full control (i.e. multiview data), additional modalities (i.e. depth and normals) and pixel perfect annotations (i.e. 3D keypoints). Research such as the seminal “Sapiens” work has further demonstrated success by finetuning foundational models trained on real data with synthetic data, to enable generalization despite limited amounts of synthetic assets.

All of those models are discriminative – they analyze real images to derive higher order semantics from it, such as keypoints, segmentations, or surface normals. On the generative side people have been considerably less successful, due to the remaining domain gap in the data generated, causing domain shifts as visible for example in “Rodin”. Recent works, such as “Cafca” or “SynShot”, have proposed to overcome this domain shift by fine-tuning on real data, typically following a three-step approach: 1) pretrain a prior on synthetic data, 2) invert sparse views into that prior, 3) employ a fine-tuning strategy to go off model. While this can produce results without domain shift, it’s of course not a fully generative model anymore, and as such has its own limitations – for example one cannot unconditionally sample from it. Looking forward we predict the development to trend towards fusing high quality 3D synthetic data with the emerging large scale image or video foundational models to overcome the domain gap for generative 3D models.

Reconstructing faithfully the 3D architecture of plants from unstructured observations is a challenging task. Plants frequently contain numerous organs, organized in branching systems in more or less complex spatial networks, leading to specific computational issues due to self-occlusion or spatial proximity between organs. Existing works either consider inverse modeling where the aim is to recover the procedural rules that allow to simulate virtual plants, or focus on specific tasks such as segmentation or skeletonization. We propose a unified approach that, given a 3D scan of a plant, allows to infer a parameterized representation of the plant. This representation describes the plant’s branching structure, contains parametric information for each plant organ, and can therefore be used directly in a variety of tasks.

Personalized anatomical digital twins are essential for medicine, computer graphics, and biomechanics, but serving internal anatomy usually requires expensive medical imaging. Instead, we can leverage the correlation between external body shape and internal structures to predict the anatomy directly from a subject’s appearance. Learning this correlation raises three key challenges: building datasets with paired observations of body shapes and internal anatomy, annotating these datasets, and learning models that capture the correlation between external and internal features. In this talk, I will showcase how we became able to predict skeleton geometry, bone location, and soft tissue distribution solely from external body shape.

Deep learning is a disruptive line of research that continues to reshape how computational problems are formulated and solved. While remarkable advances have been made in tasks involving classification, segmentation, and reconstruction, certain fundamental limitations remain – especially when handling geometric data not suited to traditional convolutional structures. This note outlines our ongoing efforts to address shape reconstruction and analysis in such scenarios, drawing inspiration from differential geometry, spectral theory, and human vision.

Deep learning architectures are designed around the notion of shift-invariance and low-dimensional latent structures. These assumptions empower convolutional neural networks (CNNs) to generalize well across a range of problems. However, for geometric structures where no natural shift-invariant coordinate system exists, standard methods often fall short. In our group, we have been exploring new avenues that combine classical geometry with modern machine learning to reconstruct and analyze shapes – even when those shapes are partially occluded, topologically noisy, or nonrigid. These include tools for shape matching, geodesic measurement, and the design of semi-supervised learning techniques rooted in axiomatic geometric principles.

One of geometry’s grand challenges is explaining the world we live in. Shape comparison, particularly of nonrigid objects, exemplifies this. Unlike rigid objects, nonrigid shapes lack a universal parameterization, complicating comparison. Over the past two decades, we have developed methods for analyzing and matching such shapes using the Gromov Hausdorff (GH) distance, which quantifies discrepancies between metric spaces. We have advanced this from theoretical abstraction to practical approximation using spectral embeddings.

Traditionally, computer vision (CV) aimed to extract geometry from images, while computer graphics (CG) sought to generate images from geometry. Today, these domains converge. A fundamental question we address is: Can neural networks help discover invariants or derive parameterizations of geometric data? To this end, we showed that the Gromov distance between metric spaces offers a powerful framework for non-rigid shape analysis.

Classical computer vision focused on “shape from X” problems: shape from shading (often solved using eikonal solvers), shape from stereo (requiring correspondence resolution), and shape from auto-stereograms. A compelling example comes from stereoscopic vision. Consider looking through a cardboard tube with one eye, while the other eye is blocked by one hand – this creates a visual illusion of a “hole” in the hand, as the brain attempts to force a false correspondence between views. Such hallucination phenomena motivated the study of auto-stereograms generation, see [2], and geometric reconstruction from such data using methods described in [1]. It motivates the concept of prior based geometry reconstruction as recently adopted and adapted in [3]. It demonstrates how prior knowledge of image formation models captured by neural network trained to solve the shape from stereo problem, can significantly improve reconstruction accuracy integrating Gaussian splatting with synthetic stereo pairs.

Recently, our attention has turned to shape invariants, especially under partial observation. We introduced the Wormhole Loss [4], designed for robust partial matching by emphasizing topological consistency. This follows our earlier work on partial shape correspondence using functional maps [5], which outperforms previous efforts such as [6, 7] by incorporating both local and global topology sensitive invariants into the learning objective.

Finally, another classical idea that should be re-examined through the lens of deep learning is edge detection. The variational model for optimal edge integration via regularized Laplacian zero-crossings [8] raises an intriguing question: can such elegant mathematical formulations be embedded into a neural network framework? If so, which applications would benefit most, and how could these formulations serve as loss functions or regularizers in modern computer vision applications?

This open discussion centers on the problem of representing and understanding the structural topology of growing plants, raising foundational questions about how plant morphology evolves over time and how it can be modeled computationally. Participants explored whether plant topology – particularly the branching structure – changes during growth, and if so, how to model this dynamic nature in a consistent and analyzable way. Analogies to human anatomy (e.g., variability in tooth roots or spinal structure) highlighted similar topological ambiguity in biological systems.

A range of modeling strategies were discussed. Neural implicit representations such as NeuralSDF offer a powerful tool for interpolating between shapes, after which a skeleton can be extracted. However, challenges arise when interpolations generate non-physical or biologically invalid intermediate states – for example, a plant structure with a fractional number of branches, or a tooth with an unrealistic root configuration. In this context, skeleton extraction and branching modeling must be grounded in biological plausibility.

Stochastic modeling was proposed as an effective alternative to rule-based procedural methods. While procedural modeling requires the system to strictly follow hand-crafted rules, stochastic models can learn from data, making them more flexible for capturing the natural variability in plant structures. Parameters of branching processes (e.g., angles, lengths, probabilities of bifurcation) can be learned directly from real plant data, enabling more realistic simulations.

There was a shift in perspective proposed: rather than thinking in purely topological terms, plant structures may be better understood as graph-like entities, where branches and junctions form a structured hierarchy. This aligns better with biological reality and allows for more intuitive comparisons between different plant forms.

To compare plant structures across stages of growth or between individuals, several mathematical tools were proposed. Diffusion distances were suggested as a more robust alternative to geodesic distances, offering smoother and more meaningful metrics for shape comparison. In cases where the structure contains holes or discontinuities, Fourier-based approaches might offer better representations. Still, the core challenge remains: how to define meaningful correspondences between topologically diverse structures, especially when interpolations between them don’t represent real physical states (e.g., morphing between different glyph forms of the letter “g”).

From an application-driven standpoint, several motivations for understanding plant topology were discussed. These include:

• $\mathrm{■}$

  Monitoring and diagnosing plant health (e.g., detecting disease via structural anomalies).

• $\mathrm{■}$

  Understanding growth patterns of individual plants versus species-level trends (nature vs. nurture).

• $\mathrm{■}$

  Predicting future states or motions of plants based on current observations.

The discussion emphasized the importance of discrete vs. continuous modeling, particularly in cases like the transition from flower to fruit, which represents a discrete structural change atop a generally continuous growth process. Mixture models or hybrid representations (continuous shapes with discrete categorical states) were considered useful for capturing this dual nature.

Drawing parallels to 3D morphable models (3DMMs) used in face reconstruction, the conversation highlighted the trade-off between model complexity and usability. While complex models may better capture biological variance, they are harder to fit and interpret. 3DMMs benefit from disentanglement and controllability, features that plant models currently lack due to the high diversity and complexity of plant morphologies.

Throughout the discussion, there was a recurring theme: the purpose of the model should drive its design. For instance, determining a person’s eye color is more efficiently done through observation than genetic analysis – a metaphor for focusing modeling efforts on efficient, goal-aligned representations. Similarly, there’s a risk in over-parameterizing models, leading to poor generalization, or under-parameterizing, resulting in loss of essential structure.

Ultimately, effective plant topology modeling must balance generative and discriminative approaches, provide biologically grounded constraints, and respect the discrete-continuous duality of natural structures. The goal is not just to simulate growth but to create meaningful, applicable models that align with real-world use cases such as agriculture, ecology, and evolutionary biology.

In order to explore the interplay between data, compute, and algorithms, it was proposed to analyze our current methods and training methodologies on constrained, well-defined tasks where perfect training data can be easily generated. By generating synthetic data without restrictions, the effect of scaling w.r.t to data and training time could more easily be disentangled, and algorithmic limitations could be identified. The discussions produced several suggestions of Computer Vision problems where shortcomings of the current models could be investigated, including maze solving, the n-queens puzzle, and Sudoku (as inspired by the previous talk of this seminar: Believe Propagation over Spatial Domains with Generative Models). A very simple example we considered is Conway’s Game of Life, which we analyzed with a quick hands-on experiment, by training a model to predict the update rule of the game. The model quickly learned the rules of the game using synthetic randomly generated boards, supervised by the true update rule. We also tested for generalization by excluding certain patterns from training. For the simple cases tested, the model was able to generalize to these unseen patterns, suggesting the network learns more than a simple look-up table operation.

Dataset propositions and ideas to be collaboratively captured:

• $\mathrm{■}$

  4D human motion dataset with a large number of markers to capture motions that cannot be captured by cameras, or by other motion capture datasets with lower number of markers that lead to smoothed motions.

• $\mathrm{■}$

  Large volumetric dataset of humans in motion with multi-people, self and object interactions.

• $\mathrm{■}$

  Multi view face dataset in both indoor and outdoor lighting conditions with marker alignment (invisible markers on the back of the head, or using sunscreen).

• $\mathrm{■}$

  MRI + volumetric dynamic or multi-pose captures.

• $\mathrm{■}$

  Large and diverse dataset that is captured across different labs with an agreed setting.

• $\mathrm{■}$

  DagMark: A 3D vision (different tasks) benchmarking challenge where people can submit datasets, evaluation schemes and metrics. Once submitted, the benchmarks are automatically compared to existing ones and added to the collection. This challenge can be hosted on a website for worldwide contributions.

Raised questions:

• $\mathrm{■}$

  Is it possible to simultaneously capture MRI scans and real-life images using a combination of MRI scanners and cameras?

• $\mathrm{■}$

  Plausibility vs accuracy: Can we design a dataset that supports both ends?

Video and Multi-View Generation Models are taking over (3D) Computer Vision. We are expecting them to model extremely complex, high-dimensional distributions. However, in many cases they spectacularly fail and collapse to hallucinations. I’d like to pose the question: is scaling up data and compute the best or only solution to solve this problem? And is it a good idea to keep up with exponential growth of distribution complexity? Bayesian inference came up with intelligent ways to simplify distribution complexity in the past. It might be a good time to explore similar ideas for continuous generative models in spatial domains. In this talk, I would introduce these discussion points and present some preliminary results we obtained in recent investigations.

The group discussed what defines a “hard problem” in computer vision and how models should be designed to solve them. While some problems benefit from large-scale “big hammer” solutions, others require specialized tools. In that context, neural networks were recognized as convenient, but also problematic in post-data regimes due to heavy data dependence, limited explainability, and robustness issues.

The discussion focused on two main issues:

• $\mathrm{■}$

  The first issue discussed was whether a “hard problem” is defined by the amount of data required to train a model to solve it, and whether any problem can be solved without the need for generalization if enough data is available. Several examples were presented, such as training NeRFs with synthetic/real data, learning geometric consistency in video models, understanding grammar in LLMs, and solving complex puzzles such as 3D mazes and Sudoku.

• $\mathrm{■}$

  The second issue discussed was the impact of explainability in models. Some participants argued that explainability is essential for humans, as the designers of these models, to understand what they are designing. Others believed that introducing parametric models might negatively affect model performance. An example discussed was self-driving cars, and whether they should adhere strictly to theoretical traffic laws or learn from actual human driving behavior.

Finally, caution was expressed regarding overly complex solutions (“big hammers”) that risk memorization instead of genuine generalization.