I’ve recently been working on a utility-focused pipeline to streamline the creation of 3D Morph Targets (Blendshapes) for facial models. While traditional manual sculpting is time-consuming, my goal is to leverage deep learning to generate identity-specific shapes based on 2D data.

The Foundation: CC0 Base Shapes

The project starts with the standard Blender CC0 Base Shapes (Male and Female). Using these as a topological anchor ensures that all generated data remains consistent and ready for standard animation pipelines.

Data Engineering: CelebA & Depth Anything

To feed the network, I’m using the CelebA dataset. The workflow looks like this:

  1. Face Cropping: Extracting high-quality facial crops.
  2. Depth Estimation: Processing these crops through Depth Anything to extract spatial information.
  3. HDF5 Storage: The results are stored in two distinct HDF5 files (Male/Female) to maintain organized identity sets.

The Architecture: ResNet18-UNet

The core of the system is a UNet with a ResNet18 backbone (single channel) designed to process depth maps. The network is tasked with predicting two primary components:

  • Pose & Orientation: Predicting facial translation and rotation using 6D Rotation representation for smoother optimization.
  • Expression Morphing: Predicting a shape delta related to the specific expression in the frame.

The Identity Solution: Trainable Morphs

A common hurdle in this process is decoupling expression from identity. My approach involves:

  • Persistent Identity Morphs: Beyond the base shape, every unique identity in the training set has its own trainable identity morph.
  • Training Loop: When the network processes a frame of “Identity A,” it applies the specific Identity A morph. This allows the system to backpropagate and refine both the universal expression weights and the specific identity deltas simultaneously.

Project Goals & Next Steps

This isn’t intended to be a “one-size-fits-all” inference model. Instead, it is a utility tool designed specifically to generate high-fidelity identity morph targets.

As a secondary “bonus,” I am analyzing the predicted expression morphs independently. The next phase involves finding the specific skeleton poses that correspond to these generated shapes, bridging the gap between vertex displacement and bone-based animation.

Key Technical Terms Used:

  • Morph Targets / Blendshapes: The 3D shapes representing different facial states.
  • 6D Rotation: A continuous representation of rotation that is much easier for neural networks to learn than Quaternions or Euler angles.
  • Trainable Identity Morph: Using an embedding-like approach to store the unique features of a person’s face.
  • Backpropagation: The method the AI uses to “learn” by adjusting the morphs based on the error between the prediction and the depth map.