The LLM reads the invoice, plain code checks it, and a person is meant to see only the doubtful ones
Turns invoices, receipts and supplier emails into validated records, and is built to send only the uncertain ones to a human.
Break an invoice, see where it goes
A supplier email, read by a regex stand-in for the LLM. Change a digit, delete a label, send it twice: the router decides between auto-approval and a person.
A small browser version with a deterministic fake model — no API key, nothing leaves your device. The real project is in the repo.
What it solves
Finance teams still type data from PDFs and emails into their accounting system by hand. It is slow, it does not scale with volume, and typos, duplicates and invoices that “don't add up” slip through.
How it works
- An LLM reads each PDF, email or scan (via OCR) and fills a fixed form. Every field comes with a confidence and a verbatim quote from the document as evidence.
- Plain rules then check the result: do the lines add up, is the total right, is it a duplicate, is the tax ID plausible. A quote that is not in the document is flagged.
- Clean records are approved automatically. An error or a low-confidence field sends the invoice to a review queue, where a person sees the values next to the highlighted source text. Each correction becomes a worked example for that vendor's next invoice.
The honest verdict
With a real free model, the 8 sample documents ended 4 approved and 4 in review, and the evidence quote for every field was found in the source, at 30–64 s per document on a rate-limited free tier. Of the 4 reviews, two were correct catches (a document that does not add up, a duplicate), one was the model leaving out a required key on the scan, and one was a phone photo with low OCR confidence. The fix for that omitted key was not exercised by the rerun, which passed on the first attempt; only a unit test covers it. A model's own confidence is not calibrated: it reported 1.00 for a till receipt's unlabeled reference and all-caps shop name, and the receipt was auto-approved, where the offline stand-in sends it to review. The values were right, and REVIEW_NEW_VENDORS=true is the safety net. Used as a vision reader, the free model reported 0.99 for every line, which gives the router no signal. Claude Haiku erred the other way and sent 3 correct documents per run to review (5 auto-approved against an ideal 8), and in 2 of 3 runs Qwen failed the phone photo by wrapping its transcription in a JSON list (fix verified by unit tests, not by a rerun). The accuracy comparison is too small and too easy to rank models, because every one, free included, was essentially at the ceiling. All test documents are fictional and generated by the author's script, so none of this shows how it copes with real invoices. With a real model there was nothing to correct, so the benefit of reviewer corrections is shown only with the offline stand-in. The Claude and Qwen numbers come through an OpenAI-compatible gateway; the native Claude and OpenAI adapters are tested against stubbed clients, not the live APIs. The repo's offline fake provider is a heuristic test double, and the browser demo is a further simplified regex version of it.
Next demoBulwark →