Avatar and Lip Sync

Lip sync is what makes a talking avatar read as listening and answering rather than playing back audio. For Frndo we set out to give a stylized "toon" character high-fidelity lip sync to synthesized speech, plus the small behaviors (eye movement, head turns, idle gestures) that make it feel present, all running in real time on a phone.

How it works

Speech audio goes through a pipeline written in C# for Unity:

1. Features. A mel spectrogram (FBANK) extractor turns each audio clip into features: mono conversion and peak normalization, resampling, overlapping Hann-windowed frames, our own Cooley-Tukey FFT, a mel filterbank and a dB scale. It reuses preallocated buffers to keep garbage collection out of the audio path.
2. Inference. A neural network we trained ourselves, running on Unity Sentis on the GPU compute backend, classifies the features into visemes (mouth shapes) and produces a timeline of start times and viseme ids.
3. Playback. Every frame, the audio source's playback position is mapped onto that timeline, and the matching mouth shape goes to the face.

The viseme set maps to the Oculus LipSync / MPEG-4 convention for compatibility with standard character rigs, without depending on the Oculus SDK itself.

The face and body

One blendshape controller handles both lip sync and emotion. Mouth-shape entries and expression entries match blendshape names by regular expression, so a newly rigged character is discovered and wired automatically, and re-wired when the avatar is swapped. Expressions can fire at random, in sequence, shuffled, or under external control. While the mouth is moving their intensity is clamped, and the final weight is the smoothed maximum of the two, so expression never fights the mouth.

Body animation runs a small state machine (idle, listening, talking) that crossfades between clips for the current state and can pick mood-specific clips when the conversation's tone is known.

The avatar system started as an experiment in expressive avatars and grew into the Frndo iPhone client. Related on-device ML research: Whisper.Sentis, OpenVoice for Unity and PiperTTS for Unity.