Voices and Avatars

An automated briefing still needs someone to deliver it. We wanted AuRA's episodes to feel produced by people, not churned out by an AI, so speech, faces and the edit around them got as much engineering as the research.

Voices

The early pipeline offered three families of text-to-speech: hosted OpenAI voices, the Windows speech voices, and Bark, an open model with very natural, expressive delivery. We loaded the full set of English Bark speaker prompts, added custom voices of our own stored as reusable resources, and let each recipe choose voices by role, so an introduction and a discussion could be read by different speakers.

Bark had one serious flaw: it drifts off script. A sentence would come back with words added, dropped or invented. Our fix, PureSpeech, runs Whisper over every generated sentence, compares the transcript with the intended text, computes a normalized error score, and regenerates any sentence outside the acceptable range. It's an adversarial check, one model listening to another, and it made Bark usable for unattended production. It has its own test suite.

The rebuilt platform makes speech a plugin like everything else: Chatterbox runs locally by default, with hosted voices and Windows speech as alternatives.

Making it sound produced

Raw speech sounds like raw speech, so the audio stage adds space and room tone to the voices and lays them over a music bed. The show's theme song was AI-generated, and the music bed could be generated at creation time with MusicGen. Every clip is normalized to the same frame size, frame rate and audio format so that title sequences, transitions, inserts and the presenter merge cleanly.

Faces

Presenters started as a library of generated portraits: dozens of men and women of different ages and backgrounds, plus abstract "brain" art and, for one show, a robot. A recipe could assign a presenter by role or voice. Later, short generated video clips replaced stills, so presenters could blink, breathe and shift in their seats between lines, in several generated rooms each.

The medical briefings took a different approach: a glowing orb, so the research assistant itself is speaking rather than a person standing in for it. A pre-rendered orb template also sped up processing, since there's no face to sync.

The AuRA Avatar Generator grew out of this need. It builds a photorealistic presenter and short presenter videos from attributes such as age, hairstyle, clothing and setting, so a presenter can be matched to an audience, from patient education to a professional conference.

Lip sync

Lip sync sits behind one plugin interface with a single job: take audio and a reference clip, produce a synced video.

• SadTalker came first, in a long-video variant that we updated for newer Python and library versions and refactored to run as a module inside the pipeline.
• MuseTalk joined next and became the default, with the engine chosen in configuration so the two could be swapped per run.
• MuseTalk 1.5 and LatentSync are the two engines in the rebuilt platform, both latent-diffusion approaches, again interchangeable.

We also studied how far MuseTalk could be pushed. For speed: keep the face-landmark cache between runs, since it never changes for a given piece of source footage; keep frames in memory instead of round-tripping through disk; compile the hot loops with Cython or Numba; and move compositing onto the GPU. For quality: add the perceptual, SyncNet-based lip-sync and adversarial losses described in the MuseTalk paper but missing from the released training code, and fine-tune on new 512×512 footage, recorded at high resolution from multiple angles, to get past the original model's 256×256 face region. The plan included memory, compute and storage estimates for training on dual high-end GPUs.

For work on real-time lip sync in our avatar apps, see Frndo's avatar and lip sync.