Unknown is a valid state
If identity or evidence is ambiguous, Verdict returns the questions that would unblock a decision instead of manufacturing a score.
//CASE STUDY / VERDICT
An AI workflow that knows when to stop.
I built Verdict to answer a practical engineering question: how do you use an LLM inside a revenue workflow without letting a plausible sentence become an untraceable CRM write?
- 60
- labeled eval cases
- 11
- staged pipeline steps
- 0
- scores below evidence floor
01 / THE PROBLEM
A prompt is not a production workflow.
A prompt-only lead scorer can produce a confident number with no evidence, invent a company fact, or silently create a duplicate record. The dangerous part is not that the output is obviously wrong; it is that it looks ready to paste into a system of record.
Evidence is an interface
Facts carry a source URL, quote, and verification status. A reviewer can inspect the grounding without trusting a hidden chain of thought.
Writes are proposals
CRM changes and outreach drafts are reviewable diffs. The system can prepare work, but a person owns the consequential action.
02 / RESPONSIBILITY MAP
The model handles language. Code owns the decision.
This boundary is the core design choice. It keeps probabilistic work useful while keeping business consequences inspectable and testable.
A
Model work
- Classify the inbound message into approved categories.
- Extract explicitly stated facts into a fixed schema.
- Judge entailment and flag instruction-like page content.
- Draft language from verified facts only.
B
Deterministic code
- Validate, normalize, resolve identity, and prevent duplicate writes.
- Open or close the evidence gate before scoring.
- Apply ICP points, vetoes, bands, and missing-information rules.
- Persist audit events, retries, approvals, and spend controls.
Engineering principle: the system never asks the model to make a decision that can be expressed as a rule and verified in a test.
03 / FAILURE ANALYSIS
The ambiguous lead that should not receive a score.
“Fieldwork Group” resolves to multiple candidate companies. A naive system picks the most similar name and continues. Verdict stops at identity resolution and creates an explicit handoff for a human.
The tempting shortcut
Name similarity looked high enough to keep going, but there was no exact email or domain match. Continuing would have attached research to the wrong account.
Identity uncertainty was treated as a score
The naive design mixed a probabilistic match with downstream qualification. That made a weak identity look like strong evidence.
Possible matches now stop the workflow
Only exact, confident matches may proceed. Possible matches propose no merge, emit no score, and return the candidate list plus unblocking questions.
The failure is now executable
The scenario is held in the guided demo and the evaluation suite. A future change cannot silently reintroduce the shortcut without a failing test.
04 / MEASURED PROOF
The claims point to an artifact.
Evaluation
A labeled suite grades outcomes, qualification bands, false scores, false refusals, and injection leakage across development and held-out cases.
57/60 latest cases passed · claude-haiku-4-5-20251001
Operations
The same Postgres-backed audit trail exposes stage latency, completion, spend, stuck jobs, and duplicate-write prevention on an inspectable operations page.
Source and CI
The repository includes the API, worker, domain rules, tests, deterministic fallbacks, and a CI workflow so the demo is reproducible rather than a staged screen.
05 / OWNERSHIP
What I built
I independently designed and implemented the system boundary, application, evaluation suite, database model, operations view, interface, and deployment path.
- FastAPI service and durable Postgres job queue
- Identity-first pipeline and evidence-sufficiency gate
- Deterministic scoring, CRM change sets, and human approval
- Evaluation harness with development and held-out splits
- Retry, rate-limit, spend-cap, audit, and observability controls
- Guided demo, integration blueprint, and recruiter-facing documentation
//THE FIVE-MINUTE ROUTE
Run one decision, then inspect the engineering behind it.