Validating the engine.
How the verification engine behind TallySeal shipped: an external ground-truth benchmark, a measured error rate with a stated direction of error, and a known-answer battery that gates every release.
The problem
TallySeal audits a store's product reviews and verifies which reviewers actually purchased the product they reviewed, by matching each review against the store's full order history: email identity, product family, and date relationship. The verdict engine is the product. If it is wrong, the merchant publishes a false claim. So the interesting engineering question was never whether the matching could be built, but how to know its error rate, in which direction it errs, and whether any change makes it better or worse.
A deliberate non-LLM core
The first real design decision was to make the verdict engine deterministic. A language model could plausibly judge "is this reviewer the same person as this buyer," but a deterministic pipeline gives something a model never can here: every verdict is reproducible and carries a mechanical reason, like no_matching_order or purchased_after_review. The pipeline is email normalization (gmail dot-folding, plus-tag stripping), bounded fuzzy tiers with edit-distance limits, product-family mapping, and date-window logic. For a product whose output a merchant will publicly stand behind, auditability beats generation. Knowing where a model belongs in a system, and where it does not, is part of the job.
It was still validated exactly the way a model would be.
The eval campaign
External ground truth, not self-labeled
The benchmark set was reviews independently marked as verified by the review platform itself, in the subset where the platform had direct knowledge: reviews it solicited from known buyer emails. Those labels came from a system with access to the transaction, not from eyeballing outputs.
Result: 99.83% agreement, zero unexplained errors
3,563 of 3,569 benchmark reviews agreed. All six disagreements traced to one boundary: purchases predating the order-history floor the engine had access to at the time.
Characterizing the misses
Agreement on the easy subset is not the whole story, so every one of the 4,154 reviews the engine declined to verify was decomposed into failure classes: email variants the normalizer should catch, same-name-different-domain candidates, typo-distance candidates, purchases of a different product family (correctly declined), and reviews with no order trace at all across four years of history. Counting every fuzzy candidate as a real buyer put the false-negative ceiling at roughly 2 to 3% of all reviews.
That number shaped the product. Exact matches say verified; fuzzy-tier matches say likely match instead of borrowing certainty they do not have. The direction of error is deliberate: wrongly verifying is far worse than wrongly declining.
Known-answer battery as a release gate
The decomposition classes became a permanent synthetic test battery, built from fixtures with no customer data in the repository: gmail-variant identities, near-miss email typos, same-family and different-family purchases, purchased-after-the-review races, undated reviews. The battery runs in CI, and a change that flips any known answer blocks the release.
Where models do the work
The same discipline carries to systems where a model is the right tool. In a production AI creative pipeline built for a marketing team, a Claude judge scores every render against brand rules before human review, a failure classifier separates transient model errors (retried automatically, never re-billed) from content refusals (surfaced, never retried), and per-batch cost accounting keeps model spend a bounded, visible number. Model output there is treated the way TallySeal treats a fuzzy match: unverified until a structural check agrees.
The transferable lesson
A decision engine, model-backed or not, needs a measured error rate against ground truth it did not generate, a stated direction-of-error policy, a regression battery that gates releases, and cost telemetry that makes its behavior visible. The cheapest time to build all of that is before the first user sees it.