Home/Work/Inference optimisation
Case study · Real Brokerage · Streamingo AI · Real estate · Sports analytics · 2022–26
Inference optimisation
Six optimisation levers applied to production vision and document pipelines, each one measured before it was kept: 3x the frames per second on identical hardware and a 200% speedup of the document pipeline.
- EmployerReal Brokerage · Streamingo AI
- RoleDeep learning engineer
- Period2022 – 2026
§ 01 — Problem
A pipeline of several models is only as fast as its slowest stage and only as cheap as its idlest GPU. Off-the-shelf FP32 models, run one request at a time, leave most of the hardware waiting. In the document platform that cost was throughput. In the sports engine it was frames per second, where detection, tracking, pose and court localisation had to move from offline processing to real time.
A GPU budget that cannot grow means the work has to. The requirement was more throughput on the same hardware, with accuracy held, and a number after every change that said whether the change was worth keeping.
§ 02 — Architecture
From trained model to served engine.
Every model goes down the same path: profile, lower precision, export and fuse, compile, re-measure, then into a serving tier where it is batched and placed.
§ 03 — How it works
One request, lever by lever.
The animation follows the latency budget of one composite request as the six levers land in order, and the throughput, utilisation and batching that follow.
Diagrams and films on this page are illustrations of the system's architecture and behaviour, drawn from my work at Real Brokerage · Streamingo AI. They are not screenshots or recordings of production systems, which remain confidential.
§ 04 — Approach
Vision pipelines
- Profile first, per stage: detection, tracking, pose and court localisation on Swin, VideoMAE and ViT backbones, each timed on its own before anything changed.
- Precision dropped where accuracy held: FP16 first, then INT8 with calibration.
- Every stage exported to ONNX, graph-fused, and compiled into a TensorRT engine.
- Inference batched across streams, so one pass through an engine serves many streams on the same GPU.
- Re-measured after each lever; anything that did not pay for itself was dropped.
LLM serving
- Fine-tuned open models served from self-hosted GPUs, with AWQ/GPTQ for the weights where accuracy held.
- Continuous batching, so requests join and leave a batch in flight instead of waiting for a fixed batch to fill.
- KV-cache reuse, so shared context is computed once.
- Graph optimisation, batched inference and quantisation together delivered the 200% pipeline speedup.
- Multi-cloud and self-hosted GPU clusters, with each model placed where its cost, latency and data-privacy profile fits.
§ 05 — Impact
The document pipeline ran 200% faster after graph optimisation, batched inference and quantisation, with GPU-backed inference using continuous batching on self-hosted GPUs for the fine-tuned open models. The video pipeline reached 3x the frames per second on identical hardware, turning offline processing into real time.
What generalises: profile before touching anything; drop precision only as far as accuracy allows; keep a lever only when the measurement says it paid for itself; and build a serving tier where each model runs at the cost and latency its task can bear.
