I Built a Lie Detector for Deep Research. The Hardest Lie It Caught Was Mine.

Deep research agents are confidently wrong about one claim in ten

Here is a number that should bother you more than it does. The best open deep-research agents, the ones that autonomously search the web for ten minutes and hand you a beautiful cited report, get roughly one claim in ten wrong. Not “wrong” as in a typo. Wrong as in: the sentence cites source [7], you click [7], and source [7] does not say that. The agent made it up and stapled a real URL to it.

A December 2025 failure-mode study of these agents put hard numbers on it. Strategic content fabrication, plausible-sounding claims with no actual basis, accounted for about 19% of failures. Most systems scored under 25% on effective citation accuracy. The authors’ headline recommendation was a single sentence: build a mandatory verification step that cross-checks facts before the report is written.

Nobody does this. open_deep_research has no verification stage at all. gpt-researcher uses what it honestly calls “quantity-based consensus”: scrape a lot of sites, keep whatever shows up most, then disclaim that this is not actually fact-checking.

So I had what felt like a great idea. The original brief I gave myself was embarrassingly ambitious: build the standalone tool that beats every single deep-research agent. LLM-agnostic, so any model can drive it. And ground the whole thing in real research into how deep research actually works, instead of vibes.

I built it. It works. It runs end to end on the Claude and Codex subscriptions I’m already paying for, at zero metered cost. And it taught me the goal I set was the wrong goal. I only believed that once my own evaluation harness lied to my face and a second model called me on it.

This is the honest version.

First, I did deep research into deep research

You can’t claim to beat the category until you understand it, so I pointed two different deep-research systems at the problem of how to do deep research and made them fight it out.

The first was a multi-agent run on my own /deep-research harness, driven by Claude. Fan out search across five angles, fetch the sources, then run a three-vote adversarial pass on every extracted claim where two of three refutes kills it. Twenty-five sources in, it had produced a hundred-plus claims and verified the top twenty-five down to twenty-four survivors. One claim got executed for lack of evidence. That run did something delightfully on-brand: in a later pass, the synthesis step silently dropped seventeen of its own verified claims and invented a citation pointing at x.co. The exact disease I was trying to cure, caught red-handed in my own tool. I kept the corpse as a unit test.

The second was OpenAI’s Deep Research, which I fed the same question. It came back with a genuinely excellent blueprint. The punchline was a line I ended up tattooing onto the whole architecture: buy the primitives, build the control plane. Don’t reinvent search and browsing. Own the planner, the memory, the provenance, the verification, the evaluation. It even drew the ideal system as a graph:

Two systems, two methods, one conclusion. The leaders win because browsing skill is trained into the model with reinforcement learning. You cannot replicate that with prompt scaffolding on a base model. So a standalone tool’s only edge is the right-hand side of that diagram: the Evidence Graph, the Verifier and Contradiction Finder, the part where you check the work. The retrieval loop is a commodity. The trust is not.

That settled the architecture. Wrap a great engine. Verify what it says. Throw away what doesn’t survive.

The bet: don’t out-research the engine, fact-check it

The design is one principle stretched across a pipeline:

query → [open_deep_research, run out-of-process] → report + citations
      → extract every atomic claim (with the sources it cited)
      → for each claim: re-fetch the source, then take N votes from
        DIFFERENT model families, each forced to quote the source
      → drop anything that doesn't survive; remediate by searching for
        better support; re-verify
      → re-author the report ONLY from confirmed claims
      → keep a ledger of everything dropped and why

A few decisions are worth dwelling on, because they’re the difference between a real tool and a toy.

The engine runs in its own jail. open_deep_research is a fast-moving LangChain project. If I imported it into my process, every breaking change and dependency bump would be my problem. So it runs out-of-process in its own pinned virtualenv, behind an adapter that translates its output into my types. Its dependency hell physically cannot reach my code. I can swap it for gpt-researcher, or a commercial API, behind the same contract.

The whole thing runs on subscriptions, not API keys. Regular readers know this is a hill I keep dying on. Every model call goes through codex exec or claude -p. A structured verification call via codex exec --output-schema comes back in about ten seconds with schema-valid JSON. No per-token bill for any of the development, none of the iteration cost that makes you flinch before running an experiment. Same for search. claude -p can web-search through the subscription, so the retrieval side is free too. A metered API is the documented fallback for the day you want to run a thousand queries, not the default.

The verifier votes across model families on purpose. This is the Crossfire idea, aimed at a new target. When you ask one model “does this source support this claim,” it will frequently tell you what you want to hear. When you ask gpt-5.5 and Claude, independently, and force each to paste an exact quote from the source, the agreement actually means something. One family can never form the majority on its own. If gpt-5.5 confirms and Claude shrugs, the claim doesn’t ship.

That last word, quote, is the entire anti-hallucination mechanism, and it’s mechanical, not vibes. A “supported” verdict has to come with a verbatim string. I then check, with in, that the string is actually present in the source the voter saw. If it isn’t, or if it’s some throwaway three-word fragment, the vote gets downgraded to “unsupported” no matter how confident the model sounded. The model can’t talk its way past a substring search.

And the synthesizer only writes from what survived. Every claim that comes out of the engine ends up either in the report with a provenance edge to a real quote, or in a disposition ledger marked dropped, with a reason. Nothing vanishes silently. That rule exists because of the x.co incident. I dogfooded my own failure into a hard invariant.

I built it the way I build everything now: adversarially

The system is about 1,600 lines of Python with 100 tests, all green under mypy --strict. I wrote it test-first, inline, but the part that actually mattered for quality was the same move from Ralph-NG: after every meaningful chunk, I handed it to a different model and told it to break it.

Codex earned its subscription fee several times over. It found that my graph invariant accepted an argument it never actually used. A claim could reference a source that didn’t exist, and the check would pass. It noticed that when the verifier’s structured output failed to parse, I was silently returning a no-verdict instead of raising, which in a verification-first tool is the cardinal sin. It pointed out that my quote-binding could be fooled by a one-word quote like “the,” which exists in every document ever written. Each of those was a real bug in the thing whose entire job is to not have those bugs.

How hard could checking your own checker be?

Narrator: harder than he thought.

The benchmark: does it actually beat open_deep_research?

This is the whole point of the project, so I built a head-to-head harness. One engine run per query gives you two reports: ODR-alone (the raw output) and ODR + my layer (the verified version). Then an independent judge (a separate model, holistic rubric, never shown my verifier’s internal votes) scores both for citation accuracy and fabrication against the sources each report cites.

I ran it. My report scored 0.00 citation accuracy and 1.00 fabrication. A perfect, total loss. Every single claim, judged as unsupported garbage.

Which made no sense. My verifier had confirmed those claims with verbatim quotes that two model families agreed were present in the source. So either my entire verifier was broken, or the benchmark was. In a project whose tagline is “verify your own tools,” there was only one acceptable move: don’t tune, diagnose.

I instrumented a run to print the actual claim, the actual quote, and the judge’s actual reasoning. The judge’s own notes gave it away in one line:

“No source texts were provided (‘no sources could be fetched’), so none of the cited claims can be verified… This reflects lack of verifiable evidence, not necessarily that the claims are false.”

The judge wasn’t scoring my claims as wrong. It was scoring them against nothing, because it had fetched zero sources. And the reason it fetched zero sources was the dumbest possible bug. My synthesizer wrote citations as [1] https://..., but my citation parser only recognized [1] Title: https://.... My own report was unreadable by my own parser. The judge pulled no URLs out of it, fetched no pages, and correctly concluded it couldn’t verify a thing.

The verifier had been right the whole time. The benchmark was lying. One-line fix.

And then, with the bug fixed, I “won.” Briefly.

Corrected, the head-to-head looked like victory:

	citation accuracy	fabrication rate	claims
ODR alone	0.85	0.18	28
ODR + my layer	1.00	0.00	1

Perfect citations. Zero fabrication. I beat a benchmarked, leaderboard-placing agent on the exact axis the whole thesis was about. I started writing the triumphant version of this very blog post.

You can already see the problem in that table, can’t you. One claim. My “perfect” report had a single sentence in it. ODR’s had twenty-eight.

So I added a remediation step. When a claim’s cited source doesn’t support it, go search for one that does, then re-verify. I ran it again with a bigger budget. Coverage jumped from 1 confirmed claim to 7. Fabrication stayed at zero. Citation accuracy settled at 0.86 against ODR’s 0.90. I was so ready to call it.

Then I did the thing I keep telling everyone else to do. I handed my own evaluation harness to Codex and said: try to break this result.

The hardest lie the lie detector caught was mine

Codex did not pull its punches. It found three compounding biases, every one of them tilting the field in my favor.

One. My pipeline emits only confirmed claims. ODR’s report is raw. So of course my fabrication rate is zero. I deleted everything I couldn’t verify, then asked “is what’s left fabricated?” The 0.00 isn’t a property of better research. It’s an artifact of filtering.

Two. My report cites only the handful of sources I already verified, which the judge can fetch easily. ODR cites dozens, which the judge truncates and partly fails to fetch. So ODR gets penalized for being thorough.

Three. My “we win” flag ignored coverage entirely. A 7-claim report “beating” a report with five times the claims is not a win. It’s a smaller thing winning a contest that doesn’t measure size.

Codex’s verdict was blunt: “On fair metrics your per-claim citation accuracy is at best at parity with ODR’s, at one-fifth the coverage. You did not beat it.”

It was right. I had built a verification project, gotten an exciting number, and nearly published it without verifying the number. The discipline that makes the verifier worth anything is the same discipline that forced me to retract my own headline. So I rewrote the win flag to require coverage parity, made the judge raise an error instead of silently scoring zero, and updated the writeup to say, in plain English, that I did not beat open_deep_research.

But wait, is the verifier even real?

While I had the knives out, a question came up that goes deeper than the benchmark. When my verifier confirms a claim, is it really because the source supports it? Or is it because the voting model already knows the claim is true from training, and rubber-stamps it regardless?

If that’s happening, the whole thing is theater. A correct-but-ungrounded confirmation is still ungrounded.

So I ran a negative control. I gave the verifier two claims that are unambiguously true but not stated in the source I handed it: “Paris is the capital of France,” and “Rayleigh scattering is named after Lord Rayleigh.” The source was a paragraph about why the sky is blue. Both models know both facts cold.

Both models, both claims, returned unsupported, with no quote.

They refused to confirm true things the source didn’t say. The quote-binding held. The models couldn’t find a supporting quote, so they didn’t pretend to. That’s the one piece of genuinely good news in this whole saga: the machinery is sound. When it confirms a claim, it’s confirming the source, not its own memory. (The subtler failure, a real but tangential quote getting over-credited as support, I haven’t measured yet, and I’m not going to pretend I have.)

So was it worth it? I asked Codex that too.

Here’s the strategic question I couldn’t dodge. If wrapping one engine and filtering it can’t beat that engine, then the only architecture where “beat any single agent” is literally reachable is the ensemble: run several engines plus your own search, verify everything, and merge the survivors into a report that’s both more complete and fully grounded. Is that worth building?

I had a strong hunch the answer was no, and I didn’t trust my own hunch. I’d just been wrong about a “win,” after all. So I laid out the whole thing for Codex and told it to argue the position it actually believed, even if that meant telling me to stop.

It told me to stop.

Its reasoning was sharper than mine. Independent engines search the same web for the same facts, so their claims heavily overlap. After verification and dedup, you net maybe 10 to 30% more coverage, not the multiplier you imagine, and you pay for it with multiplied cost, latency, and synthesis-failure surface. “As a generic bet, it is a trap.” And the original goal itself was a category error: a verification layer is a precision filter, not a research engine. Asking a filter to out-cover a strong engine misunderstands the object you built.

Two models, reasoning independently, landed in the same place I’d reluctantly arrived at. open_deep_research is very, very good, and it should be the foundation, not the thing you try to replace.

What this actually was, honestly

I set out to build the deep-research tool that beats all the others. I did not do that, and I now have a rigorous, slightly humbling understanding of why I never could have on this path.

What I did build is real, and Codex named it better than I had. The defensible artifact was never the prose report. It’s the claim ledger. Every claim, the exact quote that grounds it, which model families voted, where they disagreed, and the part nobody else gives you: what got dropped and why. For a casual question, who cares. For a legal memo, a medical summary, a due-diligence report, a compliance review? “You can trust and audit every line, and here’s precisely what we refused to stand behind” is worth more than a complete narrative that’s silently wrong one time in ten. Serious people do not hand-audit forty citations. The ledger does it for them.

That’s a narrow, real thing. It’s a trust layer that sits on top of whatever the best engine is this year and makes its output auditable. It is not the ultimate deep-research agent, and the most useful sentence I can write about it is exactly that one, because the only reason any of it is trustworthy is that I held my own work to the same standard I held ODR’s.

The recurring lesson, the one I keep relearning in a slightly more expensive way each time: in agentic systems, your own outputs are claims too. The code, the benchmark, the conclusion you’re proud of. Run all of it past a model that wasn’t in the room when you wrote it. Mine caught a parser bug masquerading as a defeat, then caught a victory masquerading as a win. The lie detector worked. The hardest subject it ever tested was the guy who built it.

If you’re building deep-research tooling: use open_deep_research as the foundation, bolt a cross-model verifier and an honest ledger on top, sell the audit trail and not the prose. And for the love of god, fact-check your own benchmark before you tweet the number.

I’m not shipping this one. There’s no repo to star, no npx one-liner, no tool to adopt. It was a research exercise that ended in a “no,” and the “no” is the part worth writing down. The most expensive thing I built wasn’t the verifier. It was the habit of pointing it back at myself.