LLM extraction that never decides a money value
A Python pipeline extracts the committed yield terms from legal PDFs with every quote verified in code. A person confirms them, and the TypeScript backend checks each payout against them.
1,342 documents through 12 templates, 99% field accuracy on a human-verified eval. Shipped October 2026.
Read how it worksShow less
Problem
Every financed asset on Dualmint comes with provenance documents that state, in legal language, the yield an investor is promised. Payouts were checked against values in our database, not against what those documents actually committed to.
Constraints
- The documents are PDFs in a dozen layouts, some scanned and some inconsistent with themselves.
- A model can misread a number or invent one, and a payout check built on a wrong number is worse than no check at all.
Design
- I grouped the 1,342 documents into 12 templates by fingerprinting their layout, so the model extracts once per template plus two spot-checks, not once per document. The whole build cost about US$8 in API calls.
- Every extracted value must come with a page and a quote. Code checks that the quote exists on that page and contains the value, and rejects anything that fails. Output is validated against a schema before it goes anywhere.
- When a document states two different values for the same term, neither the model nor the code picks one. The field stays empty and every mention goes to a person.
- An admin reviews and confirms each template's terms in a review screen. Confirmed terms are never edited in place; a changed document goes back through intake as new provenance. Ingest is idempotent, so re-running the pipeline never creates duplicates.
- On every monthly yield deposit, the TypeScript backend compares the payout with the confirmed terms and reports the result, match or not.
Evals
I labelled gold answers for all 12 templates and checked them field by field. The first run scored 90%, and most of the mismatches turned out to be mistakes in my own labels, not the model. After fixing the labels and narrowing one over-broad rule, the pipeline reached 99% field accuracy on the human-verified eval set.
Result
Shipped to production in late October 2026. All 12 templates are confirmed, and the first payouts it checked all matched the confirmed terms.
Limits
The 99% figure holds for these 12 templates. Quote verification proves a quote exists, not that it is the right clause, which is why a person confirms every template before any payout is checked against it. The model never decides a money value.
- Python
- Anthropic API
- Pydantic
- structured extraction
- LLM evals
- TypeScript
- Node.js

