Home/Work/Multi-LLM orchestration
Case study · Real Brokerage · Real estate · 2025–26
Multi-LLM orchestration infrastructure
An intent-routing layer that dispatches each request to the right model behind one interface. Fine-tuned for robust agentic workflows and RAG, it automates internal processes and saves thousands of developer-hours every week.
- EmployerReal Brokerage
- RoleArchitect and lead engineer
- Period2025 – 2026
- SectorReal estate brokerage
§ 01 — Problem
Every internal automation in the brokerage was hand-rolling its own LLM calls. Duplicated prompt logic, no routing, no fallback, no shared evaluation. Each team picked a model once and called it directly, so a change of model or prompt had to be repeated in every codebase, an outage at one provider was an outage for every process built on it, and there was no common way to say which model did a task best.
The requirement was one path to any model: a single interface that classifies what a request needs, sends it to the model that handles that task best at an acceptable cost and latency, falls back when a model is unavailable, and returns a typed, validated output.
§ 02 — Architecture
Five model families, one interface.
Requests enter through one interface. An intent router classifies and dispatches them to hosted or self-hosted models, with fallback, structured outputs and shared evaluation around all of them.
§ 03 — How it works
Requests, routed live.
The animation follows requests from four internal apps through the router to five model lanes, a hosted lane degrading and traffic falling back to a self-hosted model, and validated responses returning to the apps.
Diagrams and films on this page are illustrations of the system's architecture and behaviour, drawn from my work at Real Brokerage. They are not screenshots or recordings of production systems, which remain confidential.
§ 04 — Approach
Modelling
- A central intent-routing layer over Gemini 3.5 Pro/Flash, GPT-4o, LLaMA 3.3, Qwen 3 and Gemma, behind one interface.
- LoRA fine-tunes per task family. Gemini 3.5 Pro/Flash and GPT-4o fine-tuned for agentic workflows and RAG.
- Structured outputs by default: every response is validated against its schema before it returns to the caller.
- Shared evaluation across models and task families, so routing decisions rest on measured results.
Deployment
- Kubernetes with autoscaling across AWS and GCP (Vertex AI).
- Hosted and self-hosted models mixed per cost and latency profile.
- Graceful model fallback: when a model degrades, requests reroute to the next model for that task family and return when it recovers.
- One path for every team: the router is the default way to call an LLM from any internal process.
§ 05 — Impact
The layer became the organisation's default path to LLM automation. Internal processes that used to need their own prompt code, model client and error handling now call the router, and that saves thousands of developer-hours every week.
What generalises: one interface with routing behind it lets a model change without a code change in every caller; fallback is a property of the layer, so every automation gets it for free; and a shared evaluation set is what makes routing by intent defensible.
