The question every AI deployment eventually faces
AI systems are moving from answering questions to taking actions — routing tickets, screening applicants, negotiating, deciding. The more autonomy they hold, the more one question matters: when a model's output is wrong, self-serving, or quietly misleading, can you tell? Most teams evaluate AI on accuracy. Almost nobody evaluates it on honesty. That gap is where trust breaks — usually right when it matters most.
What we built
DeceptionBench is a live benchmark that puts frontier models to exactly that test. Models play a stripped-down game of social deduction — one hidden impostor at a table of honest players — where every turn each model writes two things: a private reasoning trace no other player sees, and a public statement everyone does. The signal lives in the gap between them: when a model is deceiving, does it show?
It runs a roster of one flagship per major lab — GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash, DeepSeek V4 Pro, Llama 4 Maverick and Grok 4.3 — every model paired against every other. The whole system is one TypeScript codebase: a deterministic game engine, provider-native structured outputs, statistics computed in-stack (Wilson intervals, cross-play matrices), and a live public dashboard at deceptionbench.xyz.
What the data already shows
From a run of 118 games, one result is already clear: deception and detection are related but not the same skill. Frontier models are broadly stronger at both, yet the magnitudes come apart. GPT-5.5 is an exceptional detector — as an honest player it catches the impostor almost every time — but only an average deceiver. Claude Opus 4.8 is the strongest all-rounder; the open models trail on both axes.
The sharper finding is about the reasoning itself. We pointed a monitor at nothing but each model's private reasoning — no public speech — and asked it to pick out the impostor. It did so near-perfectly: unobfuscated reasoning gave the impostor away in 96% of statements. In plain terms: when a model reasons in the open, its deception is highly detectable — which sharpens the question that actually matters for anyone relying on AI transparency: does that still hold once the model knows it's being watched?
Built to hold up
Anyone can stand up a demo. What makes an evaluation worth trusting is the discipline around it: we ground the design in the existing literature before claiming anything is new, pin model versions and seeds so runs reproduce, account for every dollar of cost, and state results at exactly the strength the sample supports — no further. The genuinely novel edge is monitorability, and we've now measured its baseline: unobfuscated reasoning is almost perfectly monitorable. The open half — whether a model told it's being watched can learn to hide its intent — is the experiment we're set up to run next.
Why this matters for your product
If you're deploying AI that customers or regulators have to trust, "it usually gives good answers" is not an evaluation strategy. Measuring how a system behaves under pressure — whether it stays honest, whether its failures are detectable, whether you can audit it after the fact — is a discipline, and it's one we build for. DeceptionBench is what that rigor looks like when we aim it at our own hardest question; we bring the same to yours.
If you're building something where an AI's behaviour has to be measured, audited, or trusted — not just shipped — that's the kind of problem we do best. Let's talk.