3D Morphable Models and Beyond

In this talk, I pointed out a fundamental issue in computer vision research, namely that there is a large gap between the performance of vision models on academic benchmarks and their generalization ability in real-world applications. As an illustrative example, I contrasted the outstanding performance of vision models on popular and challenging benchmarks with the still unsolved problem of detecting simple STOP signs with self-driving cars. The fundamental issue is that we assume in academic benchmarks that the training and test data are very similar (i.e. i.i.d. distributed), while autonomous systems that interact with the real-world are often confronted with data that is, in some aspect, different from what has been observed at training time (e.g. unseen illumination, context, occlusion, texture, etc.). In the second half of this talk, I discussed how advances in statistical generative models and neural rendering could potentially help to close the genralization gap. I referred to some of our recent work on integrating deep neural networks with statistical generative models in 2D [2] and 3D-aware architectures [1], which enabled machines to generalize to unseen occlusion, to perform amodal segmentation [5], and to reason about occlusion ordering [3]. I also discussed recent work where we used generative models to benchmark vision systems through adversarial examination [4], and efforts to design new datasets that focus on capturing real-world generalization [6].

Recent years have seen a massive jump in quality for the task of photorealistic novel view synthesis, mainly driven by hybrid neural rendering pipelines in which part or all of the scene representation or rendering process are optimized for final image quality using gradient descent. Our group at Google has focused on pushing further towards higher resolution, bigger scenes, and more physically accurate view synthesis, with the hopes that progress on representations and rendering methods for this underpinning task can be fruitfully transferred to many other problems in 3D vision. I will give a brief overview of our recent work on extending NeRF to perform better on large scenes, shiny objects, and noisy camera data.

We present a method for building high-fidelity animatable 3D face models that can be posed and rendered with novel lighting environments in real-time. Our main insight is that relightable models trained to produce an image lit from a single light direction can generalize to natural illumination conditions but are computationally expensive to render. On the other hand, efficient, high-fidelity face models trained with point-light data do not generalize to novel lighting conditions. We leverage the strengths of each of these two approaches. We first train an expensive but generalizable model on point-light illuminations, and use it to generate a training set of high-quality synthetic face images under natural illumination conditions. We then train an efficient model on this augmented dataset, reducing the generalization ability requirements. As the efficacy of this approach hinges on the quality of the synthetic data we can generate, we present a study of lighting pattern combinations for dynamic captures and evaluate their suitability for learning generalizable relightable models. Towards achieving the best possible quality, we present a novel approach for generating dynamic relightable faces that exceeds state-of-the-art performance. Our method is capable of capturing subtle lighting effects and can even generate compelling near-field relighting despite being trained exclusively with far-field lighting data.

The main theme of my work is to capture and to (re-)synthesize the real world using commodity hardware. It includes the modeling of the human body, tracking, as well as the reconstruction and interaction with the environment. The digitization is needed for various applications in AR/VR as well as in movie (post-)production. Teleconferencing and remote collaborative working in VR is of high interest since it is the next evolution step of how people communicate. A realistic reproduction of appearances and motions is key for such applications. Capturing natural motions and expressions as well as the photorealistic reproduction of images under novel views are challenging. With the rise of deep learning methods and, especially, neural rendering, we see immense progress to succeed in these challenges. In this talk, I will focus on the image synthesis of humans, the underlying representation of appearance, geometry, and motion to allow for explicit and implicit control over the synthesis process.

We study learning a structured latent space to represent and generate temporally and spatially dense 4D human body motion. Once trained, the proposed model generates a multi-frame sequence of dense 3D meshes based on a single point in a low-dimensional latent space. This latent motion representation can be learned in a data-driven framework that builds upon two existing lines of works. The first analyzes temporally dense skeletal data to capture the global displacement, poses and temporal evolution of the motion, while the second analyzes static densely captured human scans in 3D to represent realistic 3D human body surfaces in a low-dimensional space. Building upon the respective advantages of these two concepts allows the model to simultaneously represent temporal motion information for sequences of varying duration and detailed 3D geometry at every time instant of the motion. Experiments demonstrate that the resulting latent space is structured in the sense that similar motions form clusters in this space, and that the latent space allows to generate plausible interpolations between different actions.

3DMMs are morphable models of human faces. Existing mesh-based 3DMMs consider only a part of the human head, commonly the face region. While some also model the shape of the head, no existing 3DMM can model hair along with other features of human heads. This is partly due to the unavailability of full head data and due to the challenges in modeling hair with mesh-based representations. Can an implicit representation-based 3DMM help solve some of the limitations? What are the challenges of such models?

According to Aristotle, “the whole is greater than the sum of its parts”. This statement was adopted to explain human perception by the Gestalt psychology school of thought in the twentieth century. Here, we claim that when observing a part of an object which was previously acquired as a whole, one could deal with both partial correspondence and shape completion in a holistic manner. More specifically, given the geometry of a full, articulated object in a given pose, as well as a partial scan of the same object in a different pose, we address the new problem of matching the part to the whole while simultaneously reconstructing the new pose from its partial observation. Our approach is data-driven and takes the form of a Siamese autoencoder without the requirement of a consistent vertex labeling at inference time; as such, it can be used on unorganized point clouds as well as on triangle meshes. We demonstrate the practical effectiveness of our model in the applications of single-view deformable shape completion and dense shape correspondence, both on synthetic and real-world geometric data, where we outperform prior work by a large margin.

This talk aims to challenge long held and widely popular best practices for designing digital face processing pipelines. Specifically, nearly all face processing pipelines begin with face detection. Many systems then continue by localizing 2D facial landmarks for each detected face, a step typically used for face alignment and often also for 3D face reconstruction. Finally, depending on the application, some systems also estimate the 3D shape of each face. My talk proposes that these steps may be remnants of legacy designs from a time before effective deep learning was available, and are no longer required for many practical use cases. In fact, not only are these steps redundant, they also add unnecessary compute while introducing noise. As alternatives to these steps, I will share memory and compute efficient solutions for face detection, face alignment, and 3D face rendering. *The talk represents work done in academia, prior to joining Facebook / Meta AI and so does not represent that company in any way.

We discuss two works covering two different aspects of 3DMMs.

In the first part we present IMAvatar [1], a new method for learning implicit morphable head avatars from videos. Traditional morphable face models provide fine-grained control over expression and pose, but cannot easily capture geometric and appearance details. Neural volumetric representations approach photorealism but are hard to animate, and do not generalize well to unseen expressions. To address this gap we introduce IMAvatar, a novel method for learning head avatars with an implicit representation, directly from monocular videos. Inspired by conventional 3DMMs, IMAvatar represents the expression- and pose-related deformations via learned blendshapes and skinning fields. We employ ray tracing and iterative root-finding to locate the canonical surface intersection for each pixel. The experimental results show that our method improves geometry and covers a more complete expression space, compared to state-of-the-art methods.

In the second part we shift the focus to face appearance. We find that current diffuse albedo estimation methods are biased towards light skin tones due to (1) strongly biased priors that prefer light skin, and (2) algorithmic approaches that do not address the light/albedo ambiguity. We discuss here a solution for the latter that builds on a key observation: the full scene image contains important information about lighting that can be used for disambiguation. Our experimental results show significant improvement compared to state-of-the-art methods on albedo estimation, both in terms of accuracy as well as fairness.

Morphable models have been successfully used to generate synthetic data. Recent examples are the FaceSynthetics [1] and AGORA [2] datasets. These works show how synthetic data generation is a flexible, powerful tool to train machine learning models, which can be used on real image data captured in the wild. This session focused on analysing the strengths and limitations of current synthetic pipelines and identifying how we can better leverage morphable models to generate higher-quality data.

It was suggested that an important question is how much one should focus on photo-realism. Minor discrepancies in local pixel intensity statistics between generated images and real ones can severely harm inference. However, rendering all the components realistically in an image (people, their interactions with the scene, background etc.) is very challenging in practice; this suggests one might need to invest significant effort (including manual work from technical artists) to obtain high-quality data.

Moreover, even with high-quality synthetic data, we might not be able to adequately capture the long-tailed distributions commonly found in the real world. A potential way to identify limitations in the data could be to enable a feedback loop from the trained model, tested on new examples, back to the training data itself to highlight problematic cases. Currently, the research community lacks the tooling to generate realistic synthetic data at scale quickly. We see some efforts in this direction [3], but there is still a lot to do for humans.

In the end, morphable models can play a crucial role in enabling a “virtuous cycle” from data collection to inference – which allows us to understand the world through our models – to synthetic data generation: better inference can enable the generation of higher-quality data; in turn, higher-quality generated data can enable the development of more accurate and robust models to perceive the world.

State-of-the-art data-driven approaches to model 3D garment deformations are trained using supervised strategies that require large datasets, usually obtained by expensive physics-based simulation methods or professional multi-camera capture setups. An alternative is to use physics-based deformation models. Formulating the problem as a set of physics-based loss terms that can be used to train neural networks without precomputing ground-truth data. Can this approach be applied to more areas of morphable models and computer vision in general?

The group liked this idea seeing it as a variant of thinking about what explicit real-world knowledge we have that can be used to simplify the situation. However, it gets more complicated when attempting to include all other physical properties, such as material parameters and friction in garment modelling. No physical simulation can do that at the moment; parameters of materials that are entirely unknown, the mass of yarns, for example, and other material properties are not fully modelled. This massive gap between the real and simulation world limits the application of these approaches. This simulation gap resulted in the suggestion that these models should really be called ’physics inspired’ models as you are making a physical assumption about the world in the simulation, using a subspace of the physics.

Physics models are simplifications, so they are limited to the space that the model can represent; perhaps combining this first-principles approach with a data-driven refinement phase could be a way to go. Any explicit knowledge we can give the model aids in finding solutions with a good prior who needs the data. These approaches could help with explainability, as it is now possible to see what the model is understanding and doing to produce the effects.

Some expressed concern over how we combine physics models with neural scene representations. Understanding physics could help generalisation but the group see a challenge in combining these two. One of the benefits of implicit neural representations is that it can be done without correspondence.

Our final workshop session was reserved for an extended discussion focusing on ethics. So often, as researchers, we are focused on new functionalities and challenges. However, what we are starting to develop now, for example, technologies that identify an individual’s emotions, actions and identities, whilst enabling great opportunities, can have use cases that are extremely negative and not in the broader social interest. As leaders in this field, we know more about the possible issues than the general public and politicians. It is our responsibility to identify and predict any negative use cases and conceive of possible options to address these. Predicting negative use cases is a challenging problem, but one we need to have some answers for. To begin to address this, we split this discussion into four sections:

• $\mathrm{■}$

  Threats and ethical concerns.

• $\mathrm{■}$

  What should we do? What is our role?

• $\mathrm{■}$

  What more can our institutions do?

• $\mathrm{■}$

  Essential questions to ask ourselves when starting a project.