+ GenAI & RAG
Speech-Driven Facial Animation
Generates realistic facial animation and head motion from speech audio with a 4-stage pipeline: VAE, speech encoder, pose predictor and a Conformer-based diffusion model.
Source code is private
Overview
A deep learning pipeline that turns speech audio into realistic talking-face animation. It extracts 468 3D facial landmarks with MediaPipe and wav2vec speech features, encodes motion with a VAE, predicts head pose with a bidirectional GRU and generates smooth motion with a Conformer-based diffusion model, all synchronised at 30 FPS. Built with PyTorch with mixed-precision GPU training.
Highlights
- 4-stage pipeline: VAE motion encoder, attention-based speech encoder, BiGRU pose predictor and Conformer diffusion model
- 468 3D facial landmarks per frame with MediaPipe, synchronised with audio at 30 FPS
- wav2vec transformer features for speech understanding
- Identity preserved across generated frames (>95% FaceNet similarity)
- Mixed-precision GPU training for faster experiments