Frndo

Frndo is an AI companion you talk to by voice, with text available for links, maps and recommendations. You pick one of eight animated avatars, give it a name, and talk to it face to face. It remembers your preferences, your goals and the people in your life, so a conversation picks up where the last one left off, and you can search what it remembers and delete any of it. We designed Frndo as a supportive friend. It isn't a therapist or a romantic partner, and it encourages real-world connection: it helps find parks, restaurants and events, and nudges you to reach out to people.

Our role

We own all of Frndo's AI, end to end, and lead the teams that build everything around it:

• Lip sync. We trained the on-device model that turns Frndo's speech into mouth shapes. See Avatar and Lip Sync.
• Classifiers. We trained the classifiers that steer each conversation, from choosing which tool to use to recognizing what language someone is speaking.
• Language models. We trained the language models behind Frndo's conversation.
• Engineering. We run the teams building the iPhone app and the services behind it.

How the client is built

The app is a Unity project in C#, targeting iOS. Native iOS features go through one bridge pattern used everywhere: a Swift implementation calls the iOS API, an Objective-C++ file exposes C-callable functions, and C# P/Invoke declarations let Unity code call them.

A conversation turn runs as a loop:

1. Listen. Apple's on-device speech recognizer transcribes what you say, so your voice never leaves the phone. A small on-device Core ML model decides when you've finished your turn, rather than waiting on a fixed silence timeout.
2. Stream. The transcript goes to the conversation service, and the reply streams back with the speakable parts marked.
3. Speak. Replies are voiced a sentence at a time by a server-side neural voice model that streams Ogg Opus audio, so speech starts before the whole reply is rendered. Captions appear only for Frndo's speech, since half-corrected live transcription of the user is distracting.
4. Animate. Our own lip-sync model runs on the phone in Unity Sentis, turning each piece of audio into mouth shapes that the avatar follows as it plays. On-device sentiment analysis picks a facial expression for each sentence.
5. Repeat. Listening restarts once the avatar has finished speaking.

The avatar

Lip sync, facial expression and body animation come from our avatar system, the Unity system we first built as an experiment in expressive avatars and that grew into the Frndo client. One controller drives both visemes and emotional expressions on each avatar's blendshapes, limiting expression intensity while the mouth is speaking so the two never fight. Body animation moves between idle, listening and talking states, with mood-specific animations. Each avatar has two camera angles, full body and selfie. Switching avatars swaps the model, its voice and its animation wiring without losing the conversation history.

On-device voice research

Before settling the voice pipeline, we prototyped speech entirely on the device: OpenVoice for Unity for cloned voices with emotional variants, PiperTTS for Unity for a compact neural voice engine, and Whisper.Sentis for speech recognition inside Unity. These are research prototypes, not part of the app's voice pipeline.