Page 3 of a Halifax statement has 34 transactions. The 4-bit quantised Llama 3.2 1B returned something else.
That was the end of the on-device LLM as the primary PDF extraction route.
The spike
Phase H wired mlx-swift 0.21.2 and mlx-swift-examples into the iOS build. The appeal of on-device inference for bank statement parsing is real: no network hop, no server to maintain, and bank statement data is sensitive enough that local processing is the right default. Wiring a 1B model into the import pipeline meant the PDF extractor could run offline, with no data leaving the device.
The model was 4-bit quantised: small enough to run on a recent iPhone, recent enough to know what a tabular bank statement looks like. Two new Swift Package dependencies declared in ios/project.yml. First inference on a real Halifax statement succeeded. That was the high point.
What failed
Halifax PDFs have a rigid tabular layout: date, description, amount, balance, repeated in the same structure on every page. A human reading the document counts the rows. Ground truth on page 3 is 34 transactions.
The model did not return 34. The gap wasn’t a row off, or a formatting artefact you could fix with a normaliser. The model was fabricating transactions: inventing descriptions, merging rows, dropping entries. At a destructive rate: the kind where you can’t tell which rows are real and which are generated.
The problem isn’t recoverable at the post-processing layer. A post-processor can fix a malformed date or a spurious leading whitespace, but it cannot reconstruct a transaction the model dropped, or distinguish a fabricated description from a real one. Once the extraction is wrong, it’s wrong; no downstream check fixes that.
Two things about that failure are worth naming explicitly.
First: the model wasn’t malfunctioning. A 4-bit 1B model running inference on extracted PDF text does what it’s optimised to do: it returns plausible output. The problem is that plausibility and accuracy are different targets. For a financial importer, they have to be the same target.
Second: this failure mode is invisible without ground truth. A test that asks “does the importer return rows” passes. A test that asks “do the returned rows match the statement exactly” fails immediately. The spike caught the failure because it had exact row counts to check against. Without that, it ships.
PF-002: band-based parsing
Halifax PDFs have a consistent band structure. The columns land in the same coordinate ranges on every page, every statement. A band-based parser doesn’t read a statement as text. It reads it as a table, by coordinate.
The parser defines bands: an x-range for date, an x-range for description, x-ranges for debit, credit, and balance. It walks each content stream element and bins it by position. A coordinate either falls in a band or it doesn’t. No inference. No probability distribution. No temperature. Every ambiguity in the LLM path is gone because there are no inferences to make.
Result: 54 of 54 transactions extracted correctly. The LLM path was disabled. PF-002 is the current PDF extraction route.
The obvious objection
Halifax will change their statement format, and the band-based parser will silently break.
That’s a real risk. Three mitigations.
- The failure mode is only silent without a validation layer. A parser returning zero rows for a multi-page statement, or a row count that doesn’t match the page header’s advertised total, is a detectable signal the importer can surface.
- Halifax statement format is not a fast-moving target. Format changes are rare and typically announced. The LLM path’s hallucination rate was a continuous, ongoing accuracy problem; a format change on the deterministic path is a future maintenance event with a clear trigger.
- The LLM path is disabled, not deleted. If a larger or better-quantised model changes the accuracy picture, the Phase H spike code is still present. That test can be re-run. The decision doesn’t have to be made now.
The general principle: when the domain has a stable, known structure, a deterministic parser is a more tractable problem than a probabilistic one. The PDF layout isn’t going to surprise you at 2am with a temperature artefact.
Where the importer stands
PR #13 closes Phase I PI-003 (universal CSV) and PI-004 (universal PDF). Four banks (Monzo, Halifax, Revolut, HSBC) validated against a single parser implementation. Zero per-bank code paths remaining. The CSV route uses HeuristicMapper to reach a preset column signature first; the PDF route uses coordinate bands. Both paths share the same dedup and source-attribution layer downstream.
The dedup layer collapsed three coexisting hash schemes into one. Before PR #13, the codebase had positionHash for PDFs and separate schemes elsewhere. That’s now a single implementation. Source-document attribution via CDImportSource means every transaction carries a reference to the file it came from. That’s useful when the same statement gets reimported, or when tracing a row back to its source document.
What I’d do differently
Ground truth tests before any extraction path ships. The spike ran inference on real statements, and the failure was obvious because the test compared output against the actual statement row by row. “Does the importer return rows” is not a useful assertion. The right one: “does the importer return exactly these rows, in this order, with these amounts.”
Check the quantisation budget before committing to a model. A 4-bit 1B model is the right size for on-device inference in general. On structured tables, where every cell has a known correct value, compression artefacts become visible. A model that performs well at chat won’t perform well at exact data extraction. Different task, different failure tolerance.
Disable, don’t delete. The LLM path is off but present. When a 3B or 8B model reaches the same on-device footprint, the spike gets a second run without re-engineering the integration. The codebase stays honest about what was tried and why it stepped back. If the answer changes, the path is already there.
The Phase H spike was a week of work that produced a clear “not this” signal. That’s a useful output. The alternative, shipping the LLM extractor without exact row verification, would have been worse.
