Building an Internal LLM Gateway for Enterprise AI

Almost every enterprise AI programme we work with hits the same wall around the same time. The first two or three applications ship with direct calls to whichever foundation model API the first engineer picked. Then a fourth application needs a different model. Then finance asks who is spending what. Then a compliance review finds five services logging prompts to five different vendors. Then the model provider changes their pricing or an outage hits and there is no failover. At this point the organisation discovers, usually late, that it needed an LLM gateway a year ago.

This is our current view of what an internal LLM gateway is, why every serious enterprise AI programme ends up building or buying one, and the architecture we deploy when we build one from scratch.

What an LLM gateway actually is

An LLM gateway is a single internal service that sits between every application in the enterprise and every foundation model provider the enterprise uses. Applications call the gateway using a stable, provider-agnostic interface. The gateway handles authentication, routing, caching, rate limiting, observability, cost accounting, prompt logging, safety filters, and failover. The model providers become swappable implementations behind a common surface.

It is the same pattern as an internal API gateway, an identity broker, or a payments abstraction — familiar territory for enterprise architecture. What is new in 2026 is that the volume, cost, and strategic importance of model calls has made the absence of this layer expensive enough that engineering leaders are prioritising it explicitly rather than hoping the problem stays small.

The costs of not having one

The failure modes are consistent across the engagements where we walk in mid-crisis.

• No unified cost view. Finance cannot answer "what does AI cost us this month" without a spreadsheet, and the answer is usually thirty to fifty percent higher than what leadership believes. Cost per application, per team, per user cohort — invisible.
• No unified observability. When a downstream product misbehaves, tracing whether the cause is the model, the prompt, the context assembly, or the caller is a manual archaeological exercise across five different logging systems.
• No graceful failover. A provider outage takes down every product that depends on that provider. In 2026 this is no longer hypothetical — we have logged three provider-side incidents this year that lasted longer than most enterprises' recovery time objectives.
• Prompt sprawl. Multiple teams reinventing similar prompts, no central prompt registry, no way to A/B test a prompt change centrally, no way to enforce a safety-filter update across every caller in one deploy.
• Uncontrolled data exposure. Applications log full prompt payloads to different systems with different retention policies. PII assumptions vary. Compliance reviews take weeks per application because there is no shared boundary to reason about.
• Provider lock-in. Model-specific SDKs are embedded across the estate. Switching providers, or even switching between models within one provider, requires touching every application. Portability, which every board deck claims as a design goal, exists only on the slide.

An LLM gateway does not eliminate any of these problems automatically, but it turns each of them from a per-application problem into a single-service problem. That is the entire value proposition.

The reference architecture we deploy

Every enterprise's gateway ends up shaped slightly differently, but the components are consistent. This is what we ship when we build one from scratch.

1. A provider-agnostic HTTP interface

Applications call a single internal endpoint using a schema modelled closely on the OpenAI Chat Completions and Responses APIs, because that shape has become the industry lingua franca and every serious client library speaks it. Streaming is supported end-to-end. Tool calling is normalised across providers so applications do not care whether the underlying model is Claude, GPT, Gemini, or an open-weight model.

2. A routing layer with model aliases

Applications never name a specific provider or model version. They name an intent — reasoning-large, reasoning-fast, summarisation, classification, vision, code. The gateway resolves the alias to a concrete model at request time. Changing the underlying model for an alias is a config change deployed centrally, not a code change deployed everywhere.

3. Authentication and per-tenant quotas

Every caller authenticates with an internal service identity, typically the same identity system used for the rest of the enterprise's internal APIs. Quotas, rate limits, and spend caps are enforced per application, per team, and per environment. When an application misbehaves it does not take down the shared model budget for everyone else.

4. Prompt caching, at the gateway boundary

Both provider-native caching (Anthropic prompt caching, OpenAI cached tokens, Bedrock cache points) and gateway-native semantic caching where appropriate. In the applications we run, well-implemented caching at this layer reduces LLM spend by 25 to 60 percent for retrieval-heavy workloads. The gateway is the only place caching policy can be applied uniformly.

5. Observability, tracing, and evaluation

Every call produces a structured trace: caller identity, alias, resolved model, prompt hash, token counts input and output, latency, cache hit or miss, safety-filter verdicts, cost. Traces flow into the enterprise observability stack alongside every other production system. Sampled traces feed automated evaluation runs so regressions in a new model version are caught before they hit users.

6. Safety filters and PII handling

Input scanning for PII, injection patterns, and policy violations. Output scanning for policy compliance and hallucination detection where warranted. These live at the gateway because policy is enterprise-wide, not application-wide. When policy changes, one deploy applies it everywhere.

7. Failover and multi-provider routing

Each alias resolves to a primary model with fallbacks. When the primary provider is unavailable or breaching latency SLOs, requests route to the fallback. Applications do not know the provider changed. This is the single most important resilience investment in a mature enterprise AI programme.

8. A prompt registry

Optional but strongly recommended. System prompts, tool definitions, and structured output schemas versioned centrally, referenced by identifier rather than embedded in application code. Enables central A/B tests, staged rollouts, and one-place safety-review of every production prompt.

Build, adopt, or buy

There are three viable paths and we've deployed all three.

• Build from scratch. The right choice for enterprises with strong platform engineering, strict security requirements that off-the-shelf products do not satisfy, or unusual routing and integration needs. Expect a small platform team and three to six months to production-grade.
• Adopt an open-source foundation. LiteLLM, OpenRouter's OSS pieces, Portkey OSS, and Cloudflare AI Gateway (with limitations) are all credible starting points. The engineering cost is lower but the operational cost of running the gateway as a first-class internal service is unchanged. Expect two to three months and ongoing platform ownership.
• Buy a commercial product. Portkey, Kong AI Gateway, Cloudflare AI Gateway, and a small handful of others now offer credible enterprise features. The trade-off is your gateway policy lives partly outside your control. Best for organisations that want the pattern in production quickly and are comfortable with SaaS in this position of the stack.

The wrong choice is delaying the decision. Every month without a gateway is a month of drift in every dimension the gateway would have unified.

What good deployment looks like

• Every model call in the enterprise goes through it. Not most. Not the ones the team remembered. Every one. Enforce with network policy and audit findings, not with documentation.
• Owned by a platform team, not by any product team. The gateway is infrastructure. It needs an on-call rotation, an SLO, a roadmap, and a person accountable end-to-end.
• Latency budget under 30 milliseconds P99, cache lookups excluded. Any more and applications will route around it.
• Failure modes that degrade gracefully. The gateway must never be a single point of failure worse than the underlying providers it fronts. Local caches, aggressive timeouts, well-defined fallback behaviour.
• A dashboard leadership actually reads. Cost per team, cost per model, cache hit rate, provider mix, error rate. Publish it. This is how the political economy of AI spending gets managed.

The enterprises that will outperform on AI in 2026 are not the ones that pick the right foundation model first. They are the ones that build the infrastructure to change their mind quickly, cheaply, and safely as the model landscape evolves. The gateway is the single highest-leverage infrastructure investment for that outcome.

If your organisation is currently making LLM calls from three or more applications and cannot answer, in one query, what you spent on inference last week broken down by team — you already need this. The good news is that the pattern is well understood, the tooling has matured, and the migration from direct calls to gateway-mediated calls is one of the more tractable enterprise AI projects to execute. Start with the alias layer, get every new application on it, and migrate the legacy callers in a planned sequence over one to two quarters. It compounds from there.