Home/Work/Inference optimisation

Case study · Real Brokerage · Streamingo AI · Real estate · Sports analytics · 2022–26

Inference optimisation

Six optimisation levers applied to production vision and document pipelines, each one measured before it was kept: 3x the frames per second on identical hardware and a 200% speedup of the document pipeline.

200%Document pipeline speedup
3xFrames per second, same hardware
INT8/ FP16Precisions in production
6Levers, each measured

§ 01 — Problem

A pipeline of several models is only as fast as its slowest stage and only as cheap as its idlest GPU. Off-the-shelf FP32 models, run one request at a time, leave most of the hardware waiting. In the document platform that cost was throughput. In the sports engine it was frames per second, where detection, tracking, pose and court localisation had to move from offline processing to real time.

A GPU budget that cannot grow means the work has to. The requirement was more throughput on the same hardware, with accuracy held, and a number after every change that said whether the change was worth keeping.

§ 02 — Architecture

From trained model to served engine.

Every model goes down the same path: profile, lower precision, export and fuse, compile, re-measure, then into a serving tier where it is batched and placed.

Fig. 1Optimisation path
Each stage is profiled on its own before anything changes. Precision drops to FP16, then INT8 with calibration, with AWQ/GPTQ for LLM weights, only where accuracy holds. The graph is exported to ONNX and fused, then compiled into one TensorRT engine per stage. Every lever is re-measured and kept only if it paid for itself. In the serving tier, vision engines take batched streams, the LLM server runs continuous batching with KV-cache reuse, and each model is placed where its cost, latency and data-privacy profile fits.

§ 03 — How it works

One request, lever by lever.

The animation follows the latency budget of one composite request as the six levers land in order, and the throughput, utilisation and batching that follow.

Fig. 2Latency budget (live illustration)
Per-stage millisecond values, utilisation figures and the per-lever log are sample values, not measurements from client systems. The end-to-end ratios, 3x frames per second and a 200% pipeline speedup, are the real outcomes.
FilmCase film, 18 seconds
1280 × 720 · H.264 · no audioDownload MP4

Diagrams and films on this page are illustrations of the system's architecture and behaviour, drawn from my work at Real Brokerage · Streamingo AI. They are not screenshots or recordings of production systems, which remain confidential.

§ 04 — Approach

Vision pipelines

  • Profile first, per stage: detection, tracking, pose and court localisation on Swin, VideoMAE and ViT backbones, each timed on its own before anything changed.
  • Precision dropped where accuracy held: FP16 first, then INT8 with calibration.
  • Every stage exported to ONNX, graph-fused, and compiled into a TensorRT engine.
  • Inference batched across streams, so one pass through an engine serves many streams on the same GPU.
  • Re-measured after each lever; anything that did not pay for itself was dropped.

LLM serving

  • Fine-tuned open models served from self-hosted GPUs, with AWQ/GPTQ for the weights where accuracy held.
  • Continuous batching, so requests join and leave a batch in flight instead of waiting for a fixed batch to fill.
  • KV-cache reuse, so shared context is computed once.
  • Graph optimisation, batched inference and quantisation together delivered the 200% pipeline speedup.
  • Multi-cloud and self-hosted GPU clusters, with each model placed where its cost, latency and data-privacy profile fits.

§ 05 — Impact

The document pipeline ran 200% faster after graph optimisation, batched inference and quantisation, with GPU-backed inference using continuous batching on self-hosted GPUs for the fine-tuned open models. The video pipeline reached 3x the frames per second on identical hardware, turning offline processing into real time.

What generalises: profile before touching anything; drop precision only as far as accuracy allows; keep a lever only when the measurement says it paid for itself; and build a serving tier where each model runs at the cost and latency its task can bear.

TensorRTONNXGraph fusionINT8 / FP16AWQ / GPTQContinuous batchingKV-cache reuseBatched streamsCUDASelf-hosted GPUs

§ 06 — At a glance

A one-page summary of this case for sharing. PNG, 1080 × 1350.

Download PNG
At-a-glance infographic for inference optimisation