The Next Model Is a System: Building the Mixture-of-Models Era

We are building the next model architecture: Mixture-of-Models. It brings independent models, policies, and execution paths together as one trainable, evaluable, and deployable system.

In less than a year, vLLM Semantic Router has grown to 5,000 stars, 150+ contributors, and more than 300,000 cumulative downloads across our Hugging Face model family. We shipped three major releases—Iris, Athena, and Themis—and watched a small experiment in semantic model selection turn into a serious open-source system for multi-model inference.

Those milestones close our first chapter and give us the technical foundation and community for what comes next:

vLLM Semantic Router is becoming the training, evaluation, and inference engine for Mixture-of-Models.

Mixture-of-Models, or MoM, has been part of our system design from day 0. Our public work has so far focused on its enabling mechanism: routing. Now we are moving the abstraction boundary higher, so a system of models, policies, preferences, and execution paths can be trained, evaluated, exported, imported, and run as one model.

If vLLM made a model fast and easy to serve, vLLM Semantic Router should make many models feel like one.

Figure 1: A Mixture-of-Models turns a heterogeneous model portfolio into one model experience.

How vLLM-SR Got Here

The first vLLM Semantic Router post asked a practical question: why give simple and difficult requests the same reasoning budget? A lightweight classifier used fixed domain labels to choose between fast and reasoning paths, helping vLLM spend inference compute more selectively.

Production traffic quickly exposed the limit of that design. Domain alone could not represent privacy, safety, context, language, modality, tools, preferences, latency, and authorization. A static label also could not account for an endpoint that was cheap but overloaded, capable but remote, or unsafe to switch into midway through an agent session.

We rebuilt the classifier layer around modular model support, shared LoRA computation, Rust/Candle inference, and Go integration. We then replaced fixed classification with a Signal–Decision architecture that separated observed evidence from policy and execution. This became the spine of the next three releases.

Milestone	When	What changed
Initial release	Sep 2025	Intent-aware selection between fast and reasoning paths
v0.1 Iris	Jan 2026	Signals, decisions, and route-scoped plugins replaced fixed classification
v0.2 Athena	Mar 2026	Model selection, memory, RAG, long context, and multimodality expanded routing into an inference control system
v0.3 Themis	Jun 2026	Stateful routing, projections, replay, protocol support, session continuity, and one production configuration contract made the system operable
Fusion and Micro-Agent	Jun 2026	The router began choosing collaboration patterns, not only individual models

Figure 2: Each stage changed the unit of control: model, decision, system, session, and finally the complete model lifecycle.

Iris made routing composable. Domain, keyword, embedding, factuality, feedback, and preference signals fed explicit decisions, while safety, PII protection, caching, hallucination detection, and tool selection became route-scoped behavior. Iris also introduced the MoM model family and described vLLM-SR as “System Level Intelligence for Mixture-of-Models.”

Athena added first-class model selection, memory and RAG, a multilingual and multimodal model stack, ROCm acceleration, and an operating dashboard. The project was becoming the control system around multi-model inference, not just a classifier in front of vLLM.

Themis turned that broader system into an operable contract:

Signals become projections. Projections feed decisions. Decisions choose algorithms. Algorithms select models.

Themis added session-aware agentic routing, replayable traces, stronger protocol support, an operator console, and runtime paths across AMD ROCm, NVIDIA CUDA, Intel OpenVINO, and CPU environments. It also made a route explainable: operators can see the evidence, policy, algorithm, and physical model behind each decision.

From Signal–Decision to Workload–Router–Pool

The releases built the runtime. Two project papers explained the architecture behind it.

The white paper, Signal Driven Decision Routing for Mixture-of-Modality Models, formalized the separation between neural evidence and symbolic policy. Fast heuristics and learned classifiers turn prompts, context, identity, safety, and modality into a structured signal vector; a Boolean engine then composes those signals into auditable policy. A typed neural-symbolic DSL parses and validates that policy before compiling it into deployable configuration. When the paper was published, the system covered thirteen signal types and thirteen model-selection algorithms, with per-decision plugins for caching, RAG, memory, safety, provider handling, and response validation.

The vision paper, The Workload–Router–Pool Architecture for LLM Inference Optimization, widened the frame. It argues that three variables have to be designed together:

• Workload: chat or agent, single-turn or multi-turn, warm or cold, prefill-heavy or decode-heavy
• Router: static semantic policy, online feedback or bandit adaptation, RL-based selection, and quality-aware cascades
• Pool: homogeneous or heterogeneous accelerators, prefill/decode topology, model placement, and KV-cache management

Those variables cannot be optimized independently. Workload shape changes which routing policy works; routing policy changes the required pool size and topology; pool state changes which route is efficient. Safety and privacy cut across all three dimensions, while cost, quality, latency, and energy define the optimization frontier. The paper maps the project’s research into a 3 × 3 WRP matrix and identifies twenty-one open directions where those dimensions still need to meet.

Figure 3: The white paper defines the programmable routing engine; the vision paper connects it to workload and physical pool design.

Together, the papers made routing programmable and tied it to workload and hardware—the two foundations MoM brings under one model contract.

Meanwhile, the runtime was already moving beyond single-model selection. Fusion, ReMoM, Confidence, Ratings, and bounded Workflows let one request invoke a controlled collaboration among models. As the Micro-Agent work showed, a client can call one model name while the serving layer selects a recipe, fans out to workers, verifies or synthesizes their results, and returns one ordinary response.

First chapter	New chapter
Route a request	Build a model system
Choose a model or capability path	Train, evaluate, and execute the whole MoM
Configure runtime policy	Package a portable, versioned model artifact
Optimize a routing decision	Optimize system intelligence across quality, cost, latency, safety, and energy
Hide backend choice behind one API	Make the complete multi-model system behave like one model

Routing remains fundamental. It is how a Mixture-of-Models allocates work, applies policy, and coordinates its parts. But routing is the mechanism. The model system is the product.

Why the Model Boundary Has to Move

Today’s AI stack is fragmented, and the system boundary between its components is still missing.

Models are fragmented. Closed frontier models, open general models, domain experts, compact local models, verifiers, and multimodal models will coexist. None wins simultaneously on quality, cost, latency, trust, privacy, and domain fit.

Compute is fragmented. GPUs, CPUs, specialized accelerators, edge devices, cloud capacity, and private clusters differ in memory, kernels, availability, price, and energy use. Model choice and placement are becoming the same decision.

Location is fragmented. Inference spans cloud, data center, and edge. Privacy or residency may rule out a stronger remote model, while a local workload may still need an on-demand cloud expert.

Preference is fragmented. There is no universal “best.” Products and users make different tradeoffs among accuracy, latency, price, privacy, safety, style, and multimodality. Those choices should shape execution directly.

Before MoM, every product assembles these pieces itself. Model selection sits in SDK wrappers, provider rules in gateways, safety checks in middleware, and fallback logic in agent graphs. Two applications using the same model portfolio can still behave differently, produce different traces, and lack a shared evaluation target.

Figure 4: Before MoM, fragmented intelligence becomes application-side routing glue.

The durable bottleneck is therefore not model access. It is intelligent allocation: deciding which intelligence should run, where it should run, how much of it to spend, and whether several models should work together.

This is also an energy problem. Hardware and inference engines improve the supply side by producing more tokens per watt per dollar. MoM addresses the demand side: which semantic work deserves those tokens at all? Sending every request to the largest model wastes capacity; forcing every request onto the cheapest model wastes capability. A mature system has to know the difference.

MoM does not erase heterogeneity; it moves it behind one model boundary. The engine owns allocation, collaboration, constraints, fallbacks, and traces, while the application sees one versioned model identity and one response. The implementation can still span open and closed models, cloud and edge, and different accelerator generations.

Figure 5: With MoM, the same fragmented resources become the internal realization of one model.

What We Mean by Mixture-of-Models

A Mixture-of-Models is a versioned composite model whose engine realizes each request through a preference-conditioned, resource-bounded path across independent models and operators. It is presented to the user through one model interface and returns one attributable result.

A multi-upstream gateway can forward traffic without owning system quality. An MoM owns an objective, an evaluation contract, a reproducible composition, and the runtime that executes it.

MoM also differs from Mixture-of-Experts. MoE routes tokens among internal experts during one forward pass; MoM coordinates independent models that may differ in architecture, owner, license, modality, protocol, context window, and hardware. An MoE checkpoint can itself be one MoM component.

	Conventional model	Mixture-of-Models
Unit of intelligence	One checkpoint	A governed system of models
Specialization	Primarily encoded in weights	Composed across independent specialists
Execution	One generation path	Selection, cascade, verification, fusion, or workflow
Optimization target	One model’s quality and efficiency	The system frontier across quality, cost, latency, safety, privacy, and energy
Deployment boundary	One runtime	Cloud, data center, and edge
User contract	One model identity	One model identity

Figure 6: Selection is one MoM topology. Cascades, parallel fusion, and bounded workflows share the same model boundary.

A portable MoM therefore needs more than weights and configuration: it needs a component manifest, capability metadata, routing and collaboration recipes, policies, preferences, evaluation suites, runtime constraints, provenance, and version history.

Open checkpoints can travel with the artifact; closed models remain authenticated external references with explicit capability and policy contracts. Exporting an MoM does not make a proprietary checkpoint portable. It makes the model system reproducible.

Turn Preferences into Models

Preferences become concrete when they are published as model identities. One MoM family can offer several operating points:

Model identity	Contract
vllm-sr/mom-v1-flash	Minimize expected latency
vllm-sr/mom-v1-light	Minimize cost above a quality floor
vllm-sr/mom-v1-ultra	Maximize quality within a declared budget
vllm-sr/mom-v1-halu	Require grounding checks and fail-closed fallback
vllm-sr/mom-v1-secu	Enforce jailbreak and PII policy before execution

Each name is a versioned model contract, not a router preset. The application chooses the behavior it needs; vLLM-SR selects and coordinates the models that deliver it while preserving hard privacy, residency, authorization, and safety constraints.

To an application, the full system remains an ordinary model call:

{
  "model": "vllm-sr/mom-v1-ultra",
  "messages": [
    {"role": "user", "content": "Review this design and identify its weakest assumption."}
  ]
}

That identity may select one model, escalate through a cascade, compare parallel answers, require grounding, or run a bounded workflow—without changing the external interface, version, or response contract.

Figure 7: Preferences are published as bounded, versioned model contracts—not hidden application-side routing presets.

Four planes separate ownership:

Plane	What it owns	Foundation already in vLLM-SR	Next step
Artifact	Components, capabilities, objectives, policy, eval contract, provenance	Canonical config, model references, DSL, versioned policy	Portable MoM import/export specification
Learning	Router-owned models, preferences, outcomes, recipe improvement	Training stack, Router Learning, replay, outcome APIs	Joint training and system-level release gates
Execution	Signals, projections, decisions, selectors, loopers, plugins	Signal–Decision runtime, Fusion, ReMoM, Workflows, safety and memory	One lifecycle-aware MoM engine
Physical	Providers, model pools, accelerators, locality, cache and energy state	vLLM backends, cloud providers, ROCm, CUDA, OpenVINO, CPU	Portable placement across cloud, data center, edge, and local devices

Figure 8: A complete MoM spans four planes: artifact, learning, execution, and physical realization.

A deployment must map logical requirements onto the models and machines available in its environment. The proposal uses four objects:

1. The bundle fixes the interface, graph, policies, behavior variant, bounds, and immutable semantic assets.
2. The binding maps logical components to eligible deployments without changing the model’s decision semantics.
3. The resolution lock freezes the constituent revisions, runtimes, images, accelerators, and provider observations.
4. The run record attributes every decision, call, constraint check, cost, and outcome to the bundle, binding, and lock that produced it.

Figure 9: One stable model identity, from portable contract to attributable run.

This separation keeps portability honest. The same mom-v1-ultra can bind to ROCm, CUDA, a private CPU or NPU node, or a hybrid deployment without promising identical outputs from opaque providers. Instead, it preserves control semantics, exposes substitutions, and gives serving and evaluation the same resolved system.

vLLM-SR as the MoM Engine

Training, evaluation, and inference must share one contract; otherwise research, benchmarks, and production drift into different systems.

Training allocation, not only weights

MoM training covers router-owned embeddings, signal encoders, preference and safety models, and selectors. It also learns allocation and collaboration: which path fits a workload and budget, when a cascade should stop, how a panel should judge or synthesize, and when an agent session should switch models. Because constituents may be independent or closed, progress does not require gradients through all of them; policies, thresholds, pools, prompts, contracts, and topology can be optimized from traces and outcomes.

The target is a frontier across quality, latency, cost, safety, privacy, reliability, locality, and energy. Replay and outcomes feed production experience back into offline training without letting the hot path silently rewrite policy.

Evaluating the MoM as one model

Evaluation must score the model identity end to end; backend benchmarks are inputs, not the result. A versioned scorecard should measure routing regret, collaboration gain, recovery, session continuity, tail latency, cost, safety, privacy, and energy. It should stress provider failures, device loss, model disagreement, workload drift, and preference changes. Each declared operating point also needs its own test: flash on its latency–quality frontier, light against its quality floor, and ultra within its budget.

The scientific test is stricter than asking whether more calls improve a benchmark. Under matched active compute, can a conditional system exploit complementary strengths and failure modes better than the best fixed model? Without that control, MoM can hide brute-force scaling behind a clever graph. Evaluations must report calls, tokens, cost, latency, and energy alongside quality—and publish when composition does not help.

Figure 10: Composition gain is meaningful only under matched active compute, with quality reported alongside calls, tokens, cost, latency, and energy.

Executing intelligence at inference time

At inference time, the engine decides whether one model is enough. It may choose a local specialist, preserve a warm session, escalate through a confidence cascade, require retrieval or verification, run a Fusion panel, or execute a bounded workflow. The runtime owns the budget, topology, fallback, trace, and response contract; the application makes a normal model call.

Figure 11: MoM is a closed lifecycle: train the allocation policy, evaluate the full system, execute it, and turn outcomes into the next validated version.

One Model That Can Move

Our target is a complete MoM that can be built, exported, imported, versioned, evaluated, deployed, and invoked as a unified model. A logical specification compiles into an immutable bundle, binds to an environment, resolves the concrete deployment, and retains the same identity for serving and evaluation.

The artifact should run across developer machines, private clusters, cloud fleets, and edge environments while its physical realization changes. A specialist may resolve to an admissible local checkpoint or managed endpoint; an accelerator runtime may be replaced. If privacy makes a remote expert unavailable, the engine follows a declared fallback or abstention path. A binding cannot silently rewrite the graph, relax a guard, or turn a panel into a cascade—those changes require a new model version.

“Run on any hardware” is an architectural requirement, not a claim that every component is portable today. The project already supports paths across ROCm, CUDA, OpenVINO, and CPU. Next, hardware capability and placement become part of the MoM contract, allowing the engine to map the model system onto what is available.

The standard for the user experience is simple:

One model identity. Many models. Any hardware.

Figure 12: One logical model identity can be realized across developer, data-center, cloud, and edge hardware.

If the application needs to know which provider owns every submodel, which device runs it, or which fallback graph to execute, the abstraction has leaked.

What Changes Now

The next stage focuses on four connected areas:

1. Define a portable MoM specification. Package components, objectives, policy, preferences, evaluation, constraints, and execution semantics as one versioned artifact.
2. Close the training–evaluation–inference loop. Improve models and recipes from evaluation and replay, then ship them through reviewable, rollback-safe releases.
3. Build a heterogeneous runtime. Map one MoM across cloud, data center, and edge using hardware, locality, energy, and data boundaries as inputs.
4. Keep the model interface boring. Make an MoM as easy to import, deploy, and invoke as a single model.

Figure 13: Four connected workstreams turn Mixture-of-Models from an execution pattern into the next model architecture.

This is a research program for how independent models should specialize, compete, verify, and collaborate; how to measure the resulting system; and how one model contract can survive across devices and environments. Our mission is:

Advancing the science of intelligence across models, devices, and environments.

We will study when composition produces capabilities beyond a single checkpoint, treat placement and energy as part of intelligence, and carry the same model contract from edge to cloud and from research to production.

Build It With Us

Building Mixture-of-Models requires more than routing. The work spans model training, evaluation, serving systems, hardware, and production operations.

Iris, Athena, and Themis improved because contributors brought real workloads, added backends, trained models, published benchmarks, found failure cases, and argued for better interfaces. MoM needs the same range of work: learned allocation, preference optimization, model cooperation, energy-aware inference, portable artifacts, open evaluation, and heterogeneous runtimes.

If you work on these problems, we want to learn from your workloads and measurements. Build an operating point, add a runtime, test a collaboration recipe, or publish a case where composition fails. MoM will be stronger if its assumptions are tested in the open.

Acknowledgments

vLLM-SR has grown through work across engineering, research, and the wider ecosystem. We thank Xunzhuo Liu, Huamin Chen, Bowei He, Yankai Chen, Fuyuan Lyu, and Steve Liu for helping shape its technical and research direction. We also thank Andy Luo and Haichen Zhang for their work on ROCm enablement, router-model training, and open MoM experimentation.

The work has also been carried by FAUST, David Shrader, Yang Wu, Ramakrishnan Sathyavageeswaran, Kuntai Wu, Aayush Saini, siloteemu, Chen Wang, Yue Zhu, Senan Zedan, Yossi Ovadia, Samzong Lu, Liav Weiss, Asaad Balum, Yehudit, Noa Limoy, Marina Koushnir, Jared Wen, Abdallah Samara, Hen Schwartz, Srinivas A, Yang Zhu, Jintao Zhang, yuluo-yx, cryo, Bishen Yu, Zhijie Wang, Hao Wu, and Qiping Pan. Their code, reviews, testing, documentation, and stewardship carried the project from one release to the next.

At this milestone, the project stands at 1,734 commits and 150+ contributors. We thank collaborators at MBZUAI, McGill University, Mila, and Rice University, and the broader vLLM, AMD, Intel, Meta, Red Hat, Microsoft, Google, IBM, NVIDIA, Hugging Face, NASA, Nutanix, DaoCloud, and open-source communities. This milestone belongs to everyone who helped turn an early router into a real system.

Figure 14: Building the MoM engine is an open systems problem that needs the full model and infrastructure community.

Join us on GitHub, explore the documentation, try the MoM model family, and meet the community in the #semantic-router channel on vLLM Slack.

The first chapter taught infrastructure how to choose a model.

The next builds the model itself as a system—across models, devices, and environments.

Welcome to the Mixture-of-Models era.