Home/Work/Evaluation harness and monitoring
Case study · Real Brokerage · Real estate · 2024–26
Evaluation harness and monitoring
An offline evaluation suite and production monitoring for a five-model document pipeline. It measured the climb from 35% to 99.85% accuracy, and it gates every change before that change ships.
- EmployerReal Brokerage
- RoleAI/ML engineer
- Period2024 – 2026
- SectorReal estate brokerage
§ 01 — Problem
Five or more models run inside one document pipeline: detection, OCR, a vision-language model, reasoning LLMs and embeddings. A change to any one of them can move the end result. A new model version, a prompt edit, a fine-tune, a quantisation step: each shifts behaviour in ways a demo on a clean PDF never shows.
Without an evaluation harness, an accuracy claim is an anecdote, and a regression reaches production before anyone sees it. The requirement was a harness that scores every change against reviewed ground truth before it ships, and monitoring that keeps watching after it does.
§ 02 — Architecture
Two surfaces, one golden set.
An offline suite gates every change. Production monitoring watches what ships and feeds hard cases back into the golden set.
§ 03 — How it works
A change goes through the gate.
The animation follows one candidate adapter through the golden set, the per-field diff, the adversarial suite and the gate, then seven days of production monitoring.
Diagrams and films on this page are illustrations of the system's architecture and behaviour, drawn from my work at Real Brokerage. They are not screenshots or recordings of production systems, which remain confidential.
§ 04 — Approach
Offline
- A golden set of documents with reviewed ground truth, scored per field.
- Every change to a model, prompt or adapter runs against the golden set and is compared with the current baseline before it ships.
- An adversarial suite of hard cases: scans, rotations, handwriting, unusual layouts.
- Shared evaluation across task families, so hosted and self-hosted models are compared on the same tasks.
In production
- Every extraction carries per-field confidence. Structured outputs are validated against a schema by default.
- Drift watching on what arrives and on confidence, so a shift is seen as it happens.
- A human review queue for low-confidence cases, so nothing passes silently.
- Reviewed cases feed back into the golden set, so the suite grows with what production actually sees.
§ 05 — Impact
The climb from 35% to 99.85% extraction accuracy was certified by the harness. Every step of it was measured per field against reviewed ground truth, and every model and routing decision along the way rests on the same measured results.
Regressions are caught before production. A change that moves a field the wrong way is held at the gate, and the cases production finds hard become the next additions to the golden set.
