Multi-LLM Architecture: 6 Providers, 14 Models, 1 Codebase

When you build a serious AI product, the question isn't "which model do I use?" but "how do I make ALL models usable?" — because every provider has its own strengths, its own pricing, and the landscape keeps shifting.

This article explains how we designed the multi-LLM architecture in Assistant Core — from native SDK integration to a prompt caching strategy that cuts input-token costs by 90%.

Why Multi-LLM?#

No single model is "the best" for every task. Real-world deployment shows that:

• GPT-5.4 is strong at complex reasoning but costs $15/1M output tokens — far too expensive for FAQs
• Gemini 3.5 Flash is fast and supports native video/audio, but is sometimes weaker at complex tool calling
• Claude Sonnet 4.6 excels at code and agentic workflows with extended thinking, but has no audio/video support
• DeepSeek V4 Flash is just $0.14/1M input — the cheapest on the market, with impressive MoE quality
• Grok 4.3 from xAI is strong at realtime voice and reasoning
• MiMo V2.5 from Xiaomi — an economy model with impressive quality, and free

If you rely on a single provider, you're locked into vendor lock-in — when that provider has an outage or raises prices, your entire system feels it.

The Current Model Landscape (June 2026)#

The platform currently supports 14 LLM models across 6 providers:

Costs range over 100x (free MiMo V2.5 → GPT-5.4 at $15.00 per 1M output). Choosing the right model for the right task has a direct impact on your operating costs.

Native SDK Architecture — No Abstraction Layer#

One key decision: we don't use LangChain or any abstraction layer. Each provider uses its own native SDK.

Why? Because abstraction layers always lag behind provider updates. When Anthropic shipped prompt caching, when Gemini added thinking mode, when OpenAI added reasoning effort — the abstraction layer needed time to catch up. Native SDKs let us adopt new features immediately.

# LLMManager — Factory that creates a native SDK client per provider
class LLMManager:
    async def get_llm_instance(self, model, assistant_id):
        provider_code = model.provider.provider_code

        # OpenAI-compatible providers (OpenAI, xAI, DeepSeek)
        if provider_code in (ProviderCode.OPENAI, ProviderCode.XAI, ProviderCode.DEEPSEEK):
            return AsyncOpenAI(api_key=api_key, base_url=base_url), config

        # Anthropic — native SDK
        if provider_code == ProviderCode.ANTHROPIC:
            return AsyncAnthropic(api_key=api_key), config

        # Google — native google-genai SDK
        if provider_code == ProviderCode.GOOGLE:
            return genai.Client(api_key=api_key), config

This architecture uses 3 SDK clients to serve 6 providers:

• AsyncOpenAI — OpenAI, xAI (base_url api.x.ai), DeepSeek (base_url api.deepseek.com), Xiaomi (base_url api.xiaomi.com)
• AsyncAnthropic — Anthropic
• genai.Client — Google (Gemini)

The Streaming Dispatcher — create_stream()#

At the heart of the architecture is a single dispatcher function that decides which streaming function to call:

def create_stream(client, config, messages, tools, ...):
    """The single entry point — ChatService only ever calls this function."""
    if config.provider == ProviderCode.GOOGLE:
        return stream_gemini_native(...)
    elif config.provider == ProviderCode.ANTHROPIC:
        return stream_anthropic_native(...)
    else:
        return stream_openai_compat(...)  # OpenAI, xAI, DeepSeek

Each streaming function yields SSE events in a unified format:

{"event": "status",   "content": "Generating response..."}
{"event": "message",  "content": "Hello"}          # Text delta
{"event": "thinking", "content": "Let me think..."}  # Reasoning delta
{"event": "aborted",  "content": "Streaming aborted"}

Switching Models Mid-Conversation#

A powerful feature: a user can be chatting with GPT-5.4, switch to Claude Sonnet 4.6 mid-conversation, and Claude picks up right where things left off with the full prior context.

The trick is a Canonical Message Format: every message is stored in Redis in the standard OpenAI format.

# Canonical format stored in Redis — regardless of which provider produced it
{"role": "system", "content": "You are a helpful assistant..."}
{"role": "user", "content": "Analyze this document"}
{"role": "assistant", "content": "Based on the document...", "tool_calls": [...]}
{"role": "tool", "tool_call_id": "call_123", "content": "result"}

When the user switches models, format_converter.py handles the conversion automatically:

• OpenAI → Anthropic: system prompt split out separately, role: "tool" → role: "user" with type: "tool_result", image format conversion
• OpenAI → Gemini: builds types.Content objects, FunctionCall/FunctionResponse, strips additionalProperties (unsupported by Gemini)

Prompt Caching — Save 90% on Input Costs#

This is the optimization with the biggest impact on operating costs. Each provider has its own mechanism:

Anthropic — A 3-Breakpoint Strategy#

Anthropic allows explicit caching via cache_control markers. The platform uses 3 of the 4 available slots:

# Breakpoint 1 — System prompt (rarely changes)
create_kwargs["system"] = [{
    "type": "text",
    "text": system_prompt,
    "cache_control": {"type": "ephemeral"},  # Cache for 5 minutes
}]

# Breakpoint 2 — Tool definitions (rarely change)
tools[-1]["cache_control"] = {"type": "ephemeral"}

# Breakpoint 3 — Automatic caching (conversation history)
create_kwargs["cache_control"] = {"type": "ephemeral"}
# Auto-places a breakpoint on the last block, sliding forward as the conversation grows

Measured results:

With automatic caching, input_tokens drops to just 10–11 tokens — the entire prefix is served from cache at a 90% discount.

OpenAI — Automatic Caching#

OpenAI caches automatically once a prompt exceeds 1,024 tokens — no configuration required:

# Check the cache via usage metadata
cached_tokens = chunk.usage.prompt_tokens_details.cached_tokens
# Cache read: 50% discount on input cost

Gemini — Implicit Caching#

Gemini caches automatically via cached_content_token_count — fully implicit, no breakpoints needed.

Provider-Specific Features#

The native SDK architecture lets us tap into each provider's unique features:

Hot-Reload Models — Zero Downtime#

Admins can switch models from the dashboard without restarting the server. Model config is stored in the database, and LLMManager builds a client dynamically from the assistant's current config.

# On each request, LLMManager builds a new client based on the model in the DB
client, config = await llm_manager.get_llm_instance(
    model=assistant.current_model,    # Read from the DB in real time
    assistant_id=assistant.id,
)

# API key resolution: assistant-specific (KMS decrypt) → env default
# Lets each assistant use its own API key

This also enables multi-tenancy: each assistant (each business) can use its own model stack, API key, and configuration.

A Model-Selection Heuristic#

From real-world deployment experience, here's a simple heuristic:

1. Start with DeepSeek V4 Flash ($0.14/1M input) — the cheapest on the market, with impressive MoE quality, ideal as a default model
2. Upgrade to Claude Sonnet 4.6 when you need code generation, agentic workflows, or strong extended thinking
3. Add GPT-5.4 for complex multimodal tasks or frontier reasoning
4. MiMo V2.5 for the economy tier — free, with solid quality for simple tasks
5. Gemini 3.5 Flash for the premium mid-tier — $1.50/1M input, 1M context, thinking mode built in

And the most important rule of all: always enable prompt caching. It's the highest-ROI optimization there is — implement it once, and save on every request.