GAUNTLET

The crash-test rig for real-time voice AI.

Voice agents rarely fail by saying the wrong thing. They fail by answering two seconds late, talking over a caller who interrupts, leaving dead air, or slowing down under load. GAUNTLET dials your agent with synthetic callers over real audio, measures every turn from the caller's ear, and returns a deterministic readiness score that can fail a pull request.

What it measured

3.897 ms

The rig's own error MEASURED

Largest absolute error over 80 loopback repetitions at 200 / 500 / 1000 / 2000 ms. CI fails above 50 ms. A loopback bound, not a wide-area claim.

8 calls

Sustained concurrency MEASURED

Held in one process on a laptop, timing error p95 ≈ 7 ms; the rig saturates at 12. The aim of 10 was not met on that machine.

A vs C

Two tunings, one suite MEASURED

Same suite, same seed, mobile network profile: the tuned agent scored 100.0 (p95 786 ms), the slow-endpointing one 76.17 (p95 1,639.5 ms, dead air 15.6 %).

4 breaches

The gate failed MEASURED

Readiness −23.83, latency p95 +853 ms, time to first response +849 ms, dead air +15.6 points — the check a pull request would see.

Every figure above comes from a file in the repository: calibration/results.csv, benchmarks/rig-2026-09-11.csv, benchmarks/runs-2026-09-11/. Those runs used the bundled agent in scripted mode without provider keys, so task success was not scored in them.

The product

The Talk to agent page: an agent picker, a call orb and the conversation with each reply's answer time — **Talk to the agent.** Call the bundled restaurant agent from the browser over the same WebSocket audio protocol the synthetic callers use. Each reply shows the answer time and the agent's own hearing / thinking / speaking split.

A call detail page: paired waveforms, turn timeline, transcript and the task verdict — **Every call, taken apart.** Caller and agent on one clock, a turn timeline, an independent Speechmatics transcript, and a judge that must quote the agent's own words for each goal it accepts.

A results page showing grade C, 76.17, and the breaching metrics in red — **A score, not an opinion.** Piecewise-linear arithmetic over the measured metrics against a versioned threshold profile. Breaches first; every input and weight visible.

A comparison of the tuned run against the slow-endpointing run with a recommendation — **Two configurations, one rule.** The comparison names which side to ship and why, from documented rules rather than judgement.

The calibration page with the 3.897 ms bound and the error at each programmed delay — **The rig publishes its own error.** A measurement tool that cannot state its accuracy has produced opinions.

The GAUNTLET landing page inside the product — **The dashboard.** Guest access, so a reviewer can open a report without an account.

What it measures

Metric	Definition
Response latency p50 / p95 / p99	the caller's last voiced sample to the agent's first audio, per turn, nearest rank
Time to first response	median across calls of the first caller turn's latency
Barge-in stop time p95	interruption onset to agent speech offset; a non-yield is censored at 2,000 ms and kept
Talk-over · dead air	simultaneous speech per call; agent-side silences longer than 1.5 s
Task success	goal checklist met with verbatim turn citations, judged twice; disagreement is reported, not resolved
Degradation ratio	impaired-run latency p95 over the clean-run p95 on the same target and suite

Run it yourself

The dashboard and the live demo run locally with no external services. Python 3.12 and Node 20+:

git clone https://github.com/hamzaahmad3006/gauntlet
cd gauntlet
python -m venv .venv && .venv\Scripts\pip install -e "backend[dev]"
cd backend && ..\.venv\Scripts\python -m gauntlet.serve     # API + worker on :8000
cd frontend && npm ci && npm run dev                        # dashboard on :5173

Sign in as the local developer: the workspace arrives with two tunings of the bundled agent, so Talk to agent and a one-call demo work immediately. Add GROQ_API_KEY, ELEVENLABS_API_KEY and SPEECHMATICS_API_KEY to .env for the AI caller, natural voices and an independent transcript.

Honest scope. This page is the project's home; the interactive dashboard above runs locally or on a container you host, and is not deployed here. Impairment is applied at the application layer to the caller's outbound audio only; frame substitution approximates packet loss rather than reproducing it; the error bound is a loopback bound; task success is model inference constrained by citations. No hardware acceleration is claimed. Groq, ElevenLabs and Speechmatics have run on a live call; LiveKit is implemented and not yet verified.