Building an AI invoice processing agent with structured outputs

Most LLM demos end at "look, it extracted some text." Production ends differently — with a validated JSON object, every field typed, every total cross-checked, and a retry loop for when the model hallucinates a tax rate. That's the gap between a prompt and an agent.

This post walks through the architecture of a structured output agent — one that accepts invoices in any format, extracts the data that matters, validates it against a strict contract, and returns clean JSON or tells you exactly why it couldn't.

01 / The problem with unstructured LLM output

Ask an LLM to extract data from an invoice and you'll get something back. Sometimes it's valid JSON. Sometimes it's JSON wrapped in markdown. Sometimes it's a paragraph that describes the JSON instead of returning it. Sometimes the field names change between requests. Sometimes the arithmetic is wrong.

This is the core issue: LLMs are probabilistic text generators, not data extractors. They don't guarantee structure, they don't guarantee types, and they definitely don't guarantee arithmetic accuracy. For a one-off demo, that's fine. For a system processing thousands of invoices a month, it's a production incident waiting to happen.

Structured outputs solve this by wrapping the LLM's response in a contract — a schema that defines exactly what fields to expect, what types they must be, and what constraints they must satisfy. The model's output either conforms to the contract or gets rejected and retried.

02 / The pipeline at a glance

Every invoice that enters the system exits as one of two things: a validated JSON object conforming to a strict schema, or an explicit error report explaining what failed and why. Nothing in between.

03 / Input ingestion

Invoices don't arrive in a single format. They come as digital PDFs, scanned PDFs, photographs, email attachments, and raw text. The agent needs to handle all of them.

Digital PDFs are straightforward — text extraction libraries pull structured content directly. Scanned PDFs and photographs are different — they're images, so text extraction returns nothing. The two options are traditional OCR or sending the document to a multimodal LLM that can read it natively. The multimodal approach is increasingly the better choice — it handles layout, rotation, poor lighting, and table structures far better than rule-based OCR pipelines.

The key design decision: normalise everything to a single text representation before the LLM sees it. The extraction prompt shouldn't need to know whether the source was a PDF or a photograph. One input shape, one extraction path. This keeps the downstream logic simple and testable.

04 / The schema — your contract with the model

Before writing any extraction logic, define exactly what a processed invoice looks like. This is the architectural backbone of the entire agent.

An invoice schema covers the obvious fields — vendor details, invoice identifiers, dates, currency, line items, subtotals, tax, and total — but the real value is in the constraints. Required fields that can't be empty. Numeric fields that must be positive. Enums for payment methods so the model can't invent categories. Dates that must follow a valid format.

And critically, cross-field validations — the relationships between fields that must hold true. Each line item's amount should equal its quantity multiplied by its unit price. Line items should sum to the subtotal. Tax should be calculable from the subtotal and tax rate. The total should equal subtotal plus tax. These arithmetic relationships are where most extraction pipelines silently fail, because models are good at reading numbers off a page but bad at verifying that those numbers are internally consistent.

Why a typed schema, not a raw dictionary

A dictionary with the right keys is not validation. A typed schema (Pydantic in Python, Zod in TypeScript, or equivalent) gives you automatic type coercion, constraint enforcement, nested validation, and clear error messages when something fails. It turns "the JSON looks right" into "the JSON is right."

The schema also serves as documentation — it's simultaneously the API contract, the database migration guide, and the prompt engineering spec. When a new field is needed, you add it in one place and everything downstream updates.

05 / LLM extraction

The extraction step is where the LLM reads the invoice text and produces a JSON object matching your schema. There are two approaches.

/function calling

Most major model providers now support structured output modes — function calling, tool use, or JSON schema enforcement. You pass the schema directly as part of the API call, and the model is constrained to produce conforming output. This is the preferred approach because the constraint is enforced at the model level, not just in your prompt. The response is guaranteed to be parseable.

/prompt-based extraction

If your model doesn't support native structured output, you embed the schema in the prompt and parse the response yourself. Less reliable — models will occasionally add commentary around the JSON or format it inconsistently — but workable with defensive parsing.

The system prompt is your most important design decision

Whether using function calling or prompt-based extraction, the system prompt drives accuracy. It needs to resolve every ambiguity the model might encounter: what to do with missing fields, how to handle multiple currencies in one document, how to interpret regional date formats, whether to extract tax as a rate or an amount.

Every ambiguity you leave unresolved, the model resolves on its own — and it resolves it differently each time. The system prompt is not an instruction; it's a specification.

06 / Three layers of validation

The LLM's output passes through three validation layers, each catching a different class of error.

/layer 1 · schema validation

The raw JSON is parsed against the typed schema. This catches type mismatches, missing required fields, constraint violations, and malformed data. If parsing fails, the schema engine returns specific error messages that can be fed back to the model on retry.

/layer 2 · arithmetic consistency

Schema validation checks individual fields. It doesn't check that fields make sense together. That's business logic — and it catches the errors that matter most in production.

Do the line items sum to the stated subtotal? Does the tax calculation hold? Does the total equal subtotal plus tax? Is the due date after the invoice date? These are simple checks, but they catch a surprising number of extraction errors. Models read numbers well; they verify arithmetic poorly.

/layer 3 · deterministic correction

Some issues don't need a retry — they can be fixed in code. Rounding discrepancies from floating-point arithmetic, case normalisation on currency codes, recalculating totals from validated line items. These are deterministic corrections that don't need the model's involvement.

The philosophy: use the LLM for extraction, use code for arithmetic. Never trust a language model to add numbers correctly. Run the addition yourself and overwrite what the model produced.

07 / Retry with targeted feedback

When extraction fails validation, the agent doesn't blindly retry. It sends the specific validation errors back to the model as context for the next attempt: here's what you got wrong, fix these specific issues.

This is targeted repair, not blind repetition. The model reads its own mistakes and self-corrects. In practice, most validation failures resolve on the second attempt. A fixed retry budget (typically three attempts) prevents infinite loops — if the model can't produce valid output in that window, the invoice is flagged for human review rather than retried indefinitely.

The retry budget is a design decision, not a technical constraint. Too few retries and you reject invoices that would have succeeded on the next attempt. Too many and you burn cost and latency on invoices that were never going to parse. Three is a reasonable starting point; adjust based on your observed success rates.

08 / Confidence scoring

Every processed invoice carries metadata about how trustworthy the result is. The confidence score reflects signals like how many retry attempts were needed, how many optional fields the model couldn't find, and whether the extracted structure suggests the model saw the full document or just a partial scan.

Below a configurable threshold, the invoice gets flagged for human review. The agent doesn't pretend to have succeeded when it's uncertain — it reports its confidence and lets the system decide. This is the difference between an agent and a script: agents know what they don't know.

Start with a conservative threshold, monitor the review queue, and relax it as the system proves itself. The goal is a feedback loop where confidence scoring improves over time — not a fixed gate that never adapts.

09 / Storage

Processed invoices need to persist. The schema maps naturally to a relational store — the invoice model becomes a table, line items become a related table, and metadata sits alongside.

Two decisions that pay off later:

Store the raw validated JSON alongside the relational data. If you add fields to the schema later, you can reprocess historical invoices from the stored JSON without re-running the LLM. Keep the source of truth available for reprocessing — the same principle as storing raw text chunks alongside vectors in a RAG system.

Store the original file. When a human reviewer flags an issue, they need to see the source document side by side with the extracted data. Without the original, review is guesswork.

10 / The API layer

The agent becomes a service through a single endpoint — accept a file, detect its type, run the pipeline, return the result.

Status codes carry meaning. A successful extraction returns 200. A failed extraction — where the system worked correctly but the invoice couldn't be parsed — returns 422. The distinction matters: 500 means the system broke; 422 means the input defeated the system. Different problems require different responses from the client.

Async at scale. Single-invoice processing can be synchronous. Batch processing can't — accept the upload, return a job identifier, process in the background, and notify on completion. The pipeline is the same; the delivery mechanism changes.

11 / Testing strategy

Structured extraction agents need three categories of tests.

The happy path. Standard invoices across a range of formats, vendors, and tax structures. Verify that every field extracts correctly, types are valid, and arithmetic is consistent. Build a diverse fixture set that represents what production actually looks like.

Failure cases. Documents with missing data, blank pages, corrupted files, and contradictory numbers. Verify that the agent fails explicitly with useful error information — never silently returning bad data, and never crashing.

The hard middle. Multi-page documents, handwritten elements, mixed languages, credits and refunds, regional formatting differences. These are the cases that work in demos and break in production. They deserve dedicated test coverage because they represent the gap between "mostly works" and "production-ready."

One principle applies across all three: test structural properties, not exact values. LLM output is non-deterministic by nature. Assert that the result has the right shape, numbers fall in valid ranges, required fields are present, and arithmetic holds — not that a specific vendor name is spelled exactly one way.

12 / What transfers

The pattern here isn't invoice-specific. Any structured extraction task — contracts, receipts, medical forms, shipping documents, compliance filings — follows the same architecture: ingest → extract → validate → retry → store.

The schema changes. The business logic changes. The bones stay the same.

The principle throughout: LLMs are good at reading, bad at arithmetic, and unreliable at consistency. Build the system that compensates for that. Use the model for what it's good at — understanding layout, identifying fields, handling ambiguity in natural language — and use deterministic code for everything else.

That's what makes it an agent, not just a prompt.