Home/Work/Multi-LLM orchestration

Case study · Real Brokerage · Real estate · 2025–26

Multi-LLM orchestration infrastructure

An intent-routing layer that dispatches each request to the right model behind one interface. Fine-tuned for robust agentic workflows and RAG, it automates internal processes and saves thousands of developer-hours every week.

1000sDeveloper-hours saved every week
5Model families behind one interface
1Interface for every internal automation
2Clouds, AWS and GCP

§ 01 — Problem

Every internal automation in the brokerage was hand-rolling its own LLM calls. Duplicated prompt logic, no routing, no fallback, no shared evaluation. Each team picked a model once and called it directly, so a change of model or prompt had to be repeated in every codebase, an outage at one provider was an outage for every process built on it, and there was no common way to say which model did a task best.

The requirement was one path to any model: a single interface that classifies what a request needs, sends it to the model that handles that task best at an acceptable cost and latency, falls back when a model is unavailable, and returns a typed, validated output.

§ 02 — Architecture

Five model families, one interface.

Requests enter through one interface. An intent router classifies and dispatches them to hosted or self-hosted models, with fallback, structured outputs and shared evaluation around all of them.

Fig. 1System architecture
Every internal app calls one interface. The intent router classifies the request and dispatches it to the model with the right cost and latency profile: hosted Gemini 3.5 Pro/Flash and GPT-4o, fine-tuned for agentic workflows, or self-hosted LLaMA 3.3, Qwen 3 and Gemma with LoRA fine-tunes per task family. Retrieval supplies grounding context for RAG tasks. When a model degrades, the router falls back to the next one. Outputs are validated against a schema before they return, and a shared evaluation set scores every model on every task family.

§ 03 — How it works

Requests, routed live.

The animation follows requests from four internal apps through the router to five model lanes, a hosted lane degrading and traffic falling back to a self-hosted model, and validated responses returning to the apps.

Fig. 2Request routing (live illustration)
App names, intents, latencies and counters in the trace are sample values, not client data.
FilmCase film, 23 seconds
1280 × 720 · H.264 · no audioDownload MP4

Diagrams and films on this page are illustrations of the system's architecture and behaviour, drawn from my work at Real Brokerage. They are not screenshots or recordings of production systems, which remain confidential.

§ 04 — Approach

Modelling

  • A central intent-routing layer over Gemini 3.5 Pro/Flash, GPT-4o, LLaMA 3.3, Qwen 3 and Gemma, behind one interface.
  • LoRA fine-tunes per task family. Gemini 3.5 Pro/Flash and GPT-4o fine-tuned for agentic workflows and RAG.
  • Structured outputs by default: every response is validated against its schema before it returns to the caller.
  • Shared evaluation across models and task families, so routing decisions rest on measured results.

Deployment

  • Kubernetes with autoscaling across AWS and GCP (Vertex AI).
  • Hosted and self-hosted models mixed per cost and latency profile.
  • Graceful model fallback: when a model degrades, requests reroute to the next model for that task family and return when it recovers.
  • One path for every team: the router is the default way to call an LLM from any internal process.

§ 05 — Impact

The layer became the organisation's default path to LLM automation. Internal processes that used to need their own prompt code, model client and error handling now call the router, and that saves thousands of developer-hours every week.

What generalises: one interface with routing behind it lets a model change without a code change in every caller; fallback is a property of the layer, so every automation gets it for free; and a shared evaluation set is what makes routing by intent defensible.

Intent routingRAGLoRA fine-tuningGemini 3.5 ProGPT-4oLLaMA 3.3Qwen 3GemmaKubernetesVertex AIAWSGCP

§ 06 — At a glance

A one-page summary of this case for sharing. PNG, 1080 × 1350.

Download PNG
At-a-glance infographic for the multi-LLM orchestration infrastructure