Making sense of messy documents
A Python and LLM pipeline for turning different kinds of supply-chain documents into structured data people can use.
A few more detailsClose the notes
The challenge
Supply-chain documents arrive in many formats. Manual review made turning them into consistent, usable product data a recurring operational burden.
What I built
I designed and built the pipeline from ingestion and parsing through classification, extraction, and summarisation. A configuration-driven layer lets non-engineers add document types without changing code.
The engineering decisions
Evaluation, guardrails, and human feedback loops are part of the system. That keeps quality visible as the variety of documents grows, while reusable components keep each new type from becoming a separate implementation.
The outcome
Structured data across roughly 500 document types, replacing manual review and making document support easier to extend.
Explore: document volume and worker capacity
Change the document count and worker pool. Then reduce the model API rate limit to see why adding workers eventually stops helping.
Capacity model. Independent jobs; time ≈ count ÷ min(workers ÷ seconds per document, API limit). Omits startup time, retries, and shared database limits. Inputs are illustrative.
