PRODUCT Multilingual AI Customer Support Voice Agent
A browser-based support MVP using LiveKit, Sarvam, Python and Next.js, designed for English, Hindi and Odia conversations with reviewable tools and transcripts.
01 / The problem
Where the friction starts.
A useful multilingual support conversation needs more than speech generation: it needs grounded answers, bounded actions and a record of what actually happened.
02 / The experience
A more useful way through.
The running local Next.js application supports a mock text session with saved transcripts and tool activity. Its separate LiveKit/Python/Sarvam audio path was inspected in source but not connected during this review. Screenshots show the real local UI; the additional portfolio walkthrough is authored text.
Explores multilingual support with traceable actions. A handoff request records a request; it does not establish that a human operator connected.
03 / The workflow
- 01Choose English, Hindi or Odia in the source application
- 02Browser audio connects through LiveKit to the Python/Sarvam worker
- 03Approved tools retrieve knowledge, business hours or customer-scoped simulated orders
- 04Persist finalized transcripts, tool results and ticket or handoff requests for review
Implemented support areas in the source MVP
Technology & integration areas
Next.js / React / LiveKit / WebRTC / Python / Sarvam STT / LLM / TTS / Zod / Prisma / SQLite
04 / Current evidence
In Development
Real local UI, mock transcript persistence and tool execution verified on 4 November 2026. Web tests: 32 passed; agent tests: 24 passed. Real multilingual voice acceptance remains pending.
4 November 2026 — Real local browser support entry with language selection. The source offers a live mode; this review used only mock mode with external providers disabled.
- Source inspectedPrivate implementation excerpts awaiting final approval
- The three selected language-contract, tool-validation and reality-gate excerpts were inspected on 4 November 2026 and authorized for this review draft. Each requires explicit final owner approval before publication. No additional private code or private repository link is included.
- Browser inspectedActual local application
- The real support entry, mock interaction, saved transcript, tool activity and rejection by the voice reality gate were captured with freshly seeded fictional data. Provider credentials were unset and external browser requests blocked.
- PendingConnected multilingual voice acceptance
- A genuine continuous 30-second session, microphone and agent audio, trusted transcript and multilingual listening review remain unverified. Selecting a language or passing unit tests does not prove speech quality.
05 / My contribution
Decisions I owned.
I developed the browser support MVP and its speech, tool and persistence integrations through AI-assisted engineering. Source inspection supports the architecture and validation boundaries described here; it does not certify live language quality or production reliability.
Technical architecture
The Next.js browser UI and server routes persist calls, finalized turns and schema-validated tool results through Prisma/SQLite. Mock mode exercises a local scripted adapter and real persistence. The separate Python agent implements a LiveKit/WebRTC and Sarvam speech pipeline with en-IN, hi-IN and od-IN language contracts. That connected audio path was source-inspected, not invoked in this review. The owner authorized the selected excerpts for this draft; all three require final approval before publication.
06 / Engineering decisions
Why the system takes this shape.
- Keep voice in the browser
- LiveKit carries browser audio to a Python worker. Sarvam provides the speech-to-text, LLM and text-to-speech stages. This build has no outbound calling or SIP/PSTN integration.
- Bound tools at the application boundary
- The source checks a per-session approved-tool list and validates inputs with schemas. Knowledge search, hours, simulated order lookup, ticket creation and handoff requests produce stored tool activity.
- Separate knowledge from instructions
- Editable, active Q&A entries use language-aware fuzzy retrieval. Versioned agent instructions and the source policy define behavior; retrieved content is reference material.
- Separate simulation from voice evidence
- Finalized transcripts and lifecycle events feed a server-side reality gate. Mock or seeded conversations cannot establish a successful real voice session.
Implementation / Reviewed source
The logic behind the interface.
Selected exact source excerpts inspected on 4 November 2026. Public repositories link to the executed revision. Private-source selections are authorized for this review draft; final publication approval remains pending.
Language is part of the validated session contract
The Python call contract enumerates English, Hindi and Odia session language codes. It establishes validation structure, not measured speech quality.
services/agent/src/voice_agent/contracts.py · lines 15–19
class CallInfo(StrictModel):
id: str
roomName: str
language: Literal["en-IN", "hi-IN", "od-IN"]
isTest: boolOwner-approved local source snapshot · reviewed local working snapshot.
Private-source excerpt — final owner approval pending.
Tool calls cross a validated application boundary
A schema validates the request, the server resolves the call and its approved-tool list, and the idempotency key is checked. Only generic validation code is shown; no session data or secret values are included.
apps/web/src/server/tools.ts · lines 97–108
export async function executeTool(callId: string, raw: unknown) {
const request = toolRequestSchema.parse(raw);
const call = await db.callSession.findUnique({
where: { id: callId },
include: { configurationVersion: true },
});
if (!call) throw new Error("Call not found.");
const approved: unknown = JSON.parse(call.configurationVersion.approvedToolsJson);
if (!Array.isArray(approved) || !approved.includes(request.toolName)) throw new Error("Tool is not approved.");
const prior = await db.toolInvocation.findUnique({ where: { idempotencyKey: request.idempotencyKey } });
if (prior) {Owner-approved local source snapshot · reviewed local working snapshot.
Private-source excerpt — final owner approval pending.
Mock sessions cannot establish live voice acceptance
The gate starts by requiring live mode and trusted microphone, room and agent evidence. The remaining function also checks duration, turns, errors and transcript persistence. The local mock replay correctly stayed unpassed.
apps/web/src/server/reality-test.ts · lines 17–22
export function calculateRealityTest(evidence: RealityEvidence) {
const failures: string[] = [];
if (evidence.mode !== "live") failures.push("Session is mock mode.");
if (!evidence.microphoneReady) failures.push("Microphone verification requires both an agent-received audio track and a saved finalized customer speech turn.");
if (!evidence.roomConnectedAt) failures.push("LiveKit room connection was not verified.");
if (!evidence.agentJoinedAt) failures.push("Agent join was not verified.");Owner-approved local source snapshot · reviewed local working snapshot.
Private-source excerpt — final owner approval pending.
Verification / 4 November 2026
What actually ran.
Recorded local execution, with fixture scope and unresolved failures kept visible. These results are separate from the portfolio’s own build checks.
npm run test (web workspace)32 tests passed · 6 files- Fresh run against the reviewed local source snapshot. Tests cover the application logic within their fixture boundaries; they do not certify provider audio or speech quality.
python -m pytest (agent tests)24 tests passed- Fresh Python agent tests with external voice providers disabled. No paid calls, microphone session or external voice API usage occurred.
Start mock session → order lookup → End session → saved callMock transcript and tool result saved · voice gate NOT PASSED- Executed the real local browser UI with a fictional order. A stored get_order_status result returned in transit. The server reality gate correctly rejected mock mode as proof of voice operation.
07 / Limits & learning
What still needs proving.
- Only offline mock interaction was runtime-verified. No real LiveKit/Sarvam voice session or paid provider usage was initiated.
- English, Hindi and Odia options and language contracts exist; connected speech quality across these languages remains unverified in this review.
- Mock orders are fictional. Real CRM/ERP, payments, refunds, booking and connection to a human operator are outside the verified scope.
- Local tests do not establish production authentication, multi-tenancy, high availability or load-tested operation. The private source is not offered as a public repository.
What I learned
A convincing mock exchange cannot prove a working voice loop. Audio delivery, lifecycle events and trusted persistence need their own acceptance evidence, especially when speech quality differs across languages.
08 / Actual screenshots
From the running local application.
Captured directly from a running local application on 4 November 2026 using newly created fictional fixtures. These are actual application screens, not production activity. Open a screen to inspect it; use the arrow keys to move and Escape to close.
4 November 2026 — Real local browser support entry with language selection. The source offers a live mode; this review used only mock mode with external providers disabled.
4 November 2026 — Actual mock text interaction in the local application: a fictional order lookup returned in transit. No microphone, audio stream or paid provider was used.
4 November 2026 — Ending the mock session correctly shows Voice test not passed: mock sessions are structurally ineligible for PASS.
4 November 2026 — Persisted mock call detail with transcript, tool activity and an unpassed voice reality gate. Fictional seed data only; the resolved mock outcome does not prove real voice operation.
4 November 2026 — Actual support entry at 390px with the Odia language option selected. This proves the interface option, not Odia speech quality.
08 / Scripted walkthrough
Offline scripted simulation
Authored examples of the support flow, not recorded conversations or executed tool results. Choose a scenario in this console to inspect the sequence. All data is fictional; nothing is sent or saved.
Voice demo / V1
Support, step by step.
Fictional text, not a recorded call. No microphone, audio or live session. Tools are illustrations; nothing is executed or saved.
Waveform illustration / no audio
Explore the conversation logic.
Real voice evidence is still pending.
Can you check the status of my demo order?
- 1. Customer
- 2. Agent
- 3. Tool
- 4. Agent