Invoice field extraction that holds up at posting time
A manufacturing client in Greater Noida receives roughly 400 supplier invoices a day, about 60% as scanned PDFs. Their AP team was manually keying line items, GSTIN, invoice number, date, and taxable amounts into Tally. We built an extraction layer using a layout-aware vision model — specifically, we pass each page to Azure Document Intelligence's `prebuilt-invoice` model and store the raw JSON alongside the PDF in S3.
The hard part was not calling the API. The hard part was deciding which fields are reliable enough to auto-post and which ones route to a review queue. We got that wrong on the first pass.
The model returns a confidence score per field. Our initial logic was: if every field on the invoice clears 0.85, post it. This fell apart within a week. Confidence scores measure the model's certainty that it read the text correctly — not that the text is the right value for that field. A supplier's invoice had the HSN code printed in the GSTIN field due to a template change. The model read it with 0.97 confidence. Tally rejected the entry because the GSTIN checksum failed.
We now run three layers of validation before anything posts.
First, structural checks. GSTIN goes through the standard 15-character regex plus the checksum digit algorithm. Invoice dates must fall within the goods-receipt date ± 15 days. PO numbers must exist in the client's open-PO list, fetched from their ERP via a cached API call. These checks catch the majority of misreads without any human involvement.
Second, cross-field arithmetic. For GST invoices in India, the sum of CGST + SGST (or IGST) must equal the tax amount derived from the taxable value × rate. We compute the expected tax from the line items and compare it to the header tax field. If the difference exceeds ₹1, we flag it. Roughly 8% of invoices fail this check — sometimes because the supplier rounded differently, sometimes because the model misread a digit in a line item.
Third, supplier-specific rules. We store a per-supplier template fingerprint keyed on GSTIN. If the model's bounding boxes for key fields shift by more than 15% from the last known template, we route to review. This catches the template-change problem that bit us early on.
After these checks, about 72% of scanned invoices post without a human looking at them. The rest land in a review queue built as a simple React app showing the PDF with bounding-box overlays on the extracted fields. A reviewer sees the model's read, the validation failure reason, and a text box to correct each field. Corrections feed back into a per-supplier accuracy log, which we use to adjust thresholds.
Three fields still go to human review every time, regardless of confidence.
**Line item descriptions.** The model reads them well, but free-text descriptions vary wildly across suppliers — "MS Round Bar 12mm" versus "Mild Steel Round Bar, Diameter 12mm" — and the client's inventory system needs a specific material code. We tried an LLM-based normalization pass using GPT-4o-mini. It worked about 80% of the time but added $0.004 per invoice and roughly 400ms latency. The client decided that a human picking the material code from a dropdown was cheaper than the error rate of automated mapping plus the API cost. We dropped it.
**Bank account numbers on the payment section.** A misread here means money goes to the wrong account. The model reads these with high confidence, but the risk profile is wrong for automation. The reviewer just glances at the last four digits against a stored supplier record.
**Reverse charge flags.** These are usually a checkbox or a Y/N field, but they determine how the entire entry posts. The model gets it right, but the downstream impact of a wrong answer is significant enough that a human confirmation is worth the 3 seconds it takes.
The cost works out to roughly ₹0.40 per invoice for the Document Intelligence calls, plus our infrastructure. The client's AP team went from 9 people doing manual entry to 4 people doing review and exception handling. The 5 who moved off that work were retrained for reconciliation tasks that were previously backlog.
We learned that the model is the easy part. The validation logic, the review UI, and the discipline of knowing which fields to never trust — that is where the actual work is.