← All projects

Two AI agents behind one chat: instant replies and deep answers

Designing a chat system where a fast AI agent answers in about a second and a deeper agent works on hard questions in the background, across text chat, video calls and voice calls.

An AI companion product, with text chat, video calls and real-time voice calls, had to choose one AI model per companion. I measured both options on the real setup: a fast model starts answering in about a second but gives shallow answers to hard questions; a thinking model gives much better answers but takes 11 to 21 seconds before the first word, on every message, even "hi".

So every companion was either quick or smart, never both. I designed a way to have both.

The idea: two agents, one conversation

  • A quick agent handles every message by default: greetings, small talk, feelings, short questions, follow-ups. It answers in one to two seconds.
  • A deep agent takes the hard ones: questions about uploaded documents, comparisons, step-by-step advice, planning, web search, photos and screen sharing.

When a message needs the deep agent, the quick agent replies at once, in character ("Ooh, let me read through both properly, one sec"), and the conversation carries on. A few seconds later the deep answer arrives as a reply to the original question, with the question quoted above it. The user is never left staring at a typing indicator.

Decisions that made it work

Routing without adding delay. Clear cases (document questions, web search, images, long multi-part questions) go straight to the deep agent by simple rules, with no extra AI call. Everything else goes to the quick agent, which has a "think about this" tool it can use when a message turns out to be harder than it looks.

One context, built once. Memory, knowledge base, uploaded documents, conversation summary and instructions are assembled once per message and handed to whichever agent answers. The second agent adds no extra lookups, and nothing that already worked (credit checks, safety checks, memory) is bypassed.

Answers that never get lost. The deep agent runs as a background job that survives server restarts and deployments and retries on a backup model if the first attempt fails. An answer always arrives unless the user cancels it, even if they've gone offline; it's waiting in the chat next time.

Several questions at once, nothing mixed up. Each open question gets its own answer, linked to its own question, so a user can ask three things in a row and get three correctly matched replies. A sensible cap (five open questions) keeps it under control without ever limiting small talk.

Changing your mind. "Actually, forget the contracts" cancels that question. The user pays only for the AI work already done, because the AI provider charges for it; failed attempts that get retried are never charged twice.

Voice calls. In real-time speech-to-speech calls (Grok), the voice model already acts as the quick agent and calls the server for memory lookups. A memory lookup is short enough to wait for; a deep answer isn't, and waiting would leave the call silent. So the new tool returns immediately, the voice keeps talking, and when the answer is ready it's spoken at the next natural pause, never over the user, with the full version written in the chat. Video-call avatars work the same way.

Safe rollout. Everything sits behind one admin switch. Off means the product behaves exactly as before, so it can ship dark, be tested by the team, then turned on for users and compared on real reply times.

What I took from it

Most "the AI is slow" problems are design problems, not model problems. Users don't mind waiting for a good answer; they mind waiting with nothing happening. Answer instantly with what you can, do the slow work in the background, and make sure it can't get lost.