Summary
The workhorse enterprise application: PDFs in, structured data out, with measured accuracy. Every AI consulting engagement is a variation on this lab.
Objectives
- 01Design a pipeline that handles messy real-world PDFs.
- 02Extract structured data with a defined schema.
- 03Measure accuracy against ground truth.
- 04Handle OCR, tables, and multi-page documents.
Key Ideas
- OCR quality dominates every downstream number.
- Per-field metrics reveal what the aggregate accuracy score hides.
Lab
Build a pipeline that ingests 100+ real PDFs from one domain (invoices, contracts, medical records — with permission, or public datasets like SEC filings). Extract a defined structured schema (10+ fields including at least one table). Handle OCR for scanned pages. Report field-level precision and recall against a hand-labeled ground truth of 20 documents. Include failure taxonomy: which fields fail, on which document types, why.
Deliverables
- — Pipeline code and deployed endpoint.
- — Ground truth labels for 20 documents.
- — Accuracy report per field with failure taxonomy.
- — Runbook for adding a new field to the schema.
Rubric
- — Realism — real messy documents, not clean synthetic PDFs.
- — Rigor — hand-labeled ground truth, per-field metrics.
- — Failure honesty — the taxonomy is specific and actionable.
- — Extensibility — adding a new field is documented and doable.