Home/Work/Evaluation harness and monitoring

Case study · Real Brokerage · Real estate · 2024–26

Evaluation harness and monitoring

An offline evaluation suite and production monitoring for a five-model document pipeline. It measured the climb from 35% to 99.85% accuracy, and it gates every change before that change ships.

99.85%Accuracy certified, from 35%
5+Models under one harness
2Surfaces · offline suite · production monitoring
GatedEvery change, before it ships

§ 01 — Problem

Five or more models run inside one document pipeline: detection, OCR, a vision-language model, reasoning LLMs and embeddings. A change to any one of them can move the end result. A new model version, a prompt edit, a fine-tune, a quantisation step: each shifts behaviour in ways a demo on a clean PDF never shows.

Without an evaluation harness, an accuracy claim is an anecdote, and a regression reaches production before anyone sees it. The requirement was a harness that scores every change against reviewed ground truth before it ships, and monitoring that keeps watching after it does.

§ 02 — Architecture

Two surfaces, one golden set.

An offline suite gates every change. Production monitoring watches what ships and feeds hard cases back into the golden set.

Fig. 1Harness architecture
A candidate change is scored per field against the golden set and the adversarial suite, compared with the current baseline, and held or shipped at the gate. In production, outputs are schema-validated and carry per-field confidence; low-confidence cases and drift route to a human review queue, and reviewed cases return to the golden set.

§ 03 — How it works

A change goes through the gate.

The animation follows one candidate adapter through the golden set, the per-field diff, the adversarial suite and the gate, then seven days of production monitoring.

Fig. 2Harness trace (live illustration)
Scores, counts, thresholds and the run itself are sample values, not client data.
FilmCase film, 23 seconds
1280 × 720 · H.264 · no audioDownload MP4

Diagrams and films on this page are illustrations of the system's architecture and behaviour, drawn from my work at Real Brokerage. They are not screenshots or recordings of production systems, which remain confidential.

§ 04 — Approach

Offline

  • A golden set of documents with reviewed ground truth, scored per field.
  • Every change to a model, prompt or adapter runs against the golden set and is compared with the current baseline before it ships.
  • An adversarial suite of hard cases: scans, rotations, handwriting, unusual layouts.
  • Shared evaluation across task families, so hosted and self-hosted models are compared on the same tasks.

In production

  • Every extraction carries per-field confidence. Structured outputs are validated against a schema by default.
  • Drift watching on what arrives and on confidence, so a shift is seen as it happens.
  • A human review queue for low-confidence cases, so nothing passes silently.
  • Reviewed cases feed back into the golden set, so the suite grows with what production actually sees.

§ 05 — Impact

The climb from 35% to 99.85% extraction accuracy was certified by the harness. Every step of it was measured per field against reviewed ground truth, and every model and routing decision along the way rests on the same measured results.

Regressions are caught before production. A change that moves a field the wrong way is held at the gate, and the cases production finds hard become the next additions to the golden set.

Golden setsPer-field scoringRegression gatesAdversarial suiteSchema validationConfidence routingDrift monitoringHuman review queue

§ 06 — At a glance

A one-page summary of this case for sharing. PNG, 1080 × 1350.

Download PNG
At-a-glance infographic for the evaluation harness and monitoring