AI Gateway: Routing, Cost Controls, and Observability

Treat the gateway as a control boundary, not a magic model improver. Product services still own the user experience, prompts still need evaluation, and model providers still generate tokens. The gateway owns the crossing point: identity, provider credentials, routing policy, request logging, budget checks, and safe fallback behavior.

The running example is support-triage. The app sends { tenantId, ticketId, severityHint, promptVersion } to the gateway. Success means the gateway returns a triage label plus a trace id. The intentional failure is a per-tenant budget cap: the gateway must reject or downgrade the request before it leaks provider keys or spends unplanned money.

The gateway path should be boring enough to debug. support-triage authenticates to the gateway, sends a typed task request, and receives either a model result or a typed refusal. Inside the gateway, policy runs before the provider call: tenant allowed, route allowed, estimated cost inside budget, prompt version known, and sensitive fields redacted or denied.

Only after those checks does the gateway choose a provider and model. The response comes back with a normalized shape so product code does not depend on every provider's error vocabulary. That normalization is useful, but it has a cost: if the gateway hides too much detail, teams cannot diagnose provider-specific latency, safety refusals, context-limit errors, or rate-limit behavior.

Budget control is not only a finance feature. It is a reliability feature because runaway loops, retries, and oversized context windows can turn a small incident into noisy provider traffic. Put the cap in the gateway so every service uses the same accounting boundary before the model call happens.

Policy should be boring and testable: who is calling, what task is requested, which data class is present, which route is allowed, how much budget remains, and whether the request needs approval. The gateway can downgrade from a premium model to a cheaper one, reject with budget_exceeded, or queue for human review. What it should not do is silently switch models in a way that changes user-visible quality without a trace.

Gateway logs are operational evidence, not quality by themselves. Latency, tokens, cost, route, and provider status explain how the request ran. Evals explain whether the answer was good enough. The two become powerful together when every eval failure links back to the prompt version, retrieved context ids, route decision, and model output that produced it.

For support-triage, keep a small golden set of tickets with expected labels and escalation rules. Before changing routes or models, replay the set through the gateway and compare quality, latency, and cost per ticket. A cheaper model is only a win if it preserves the pass threshold for the ticket mix you actually serve.

Trigger. A release accidentally increases maxOutputTokens for support-triage, and a provider also starts returning intermittent 429s. Symptom. Latency rises, monthly spend velocity spikes, and some tickets receive generic fallback labels. Root mechanism. Direct callers would each retry differently, but the gateway sees all traffic at the model boundary: estimated cost, rate-limit responses, route decisions, and fallback frequency.

Recovery: freeze the expensive prompt version, lower the route to a cheaper model for low-severity tickets, preserve high-severity tickets for the stronger route, and return provider_unavailable when no safe fallback exists. Prevention: add a spend-velocity alert, a per-task max-token policy, and a regression case that fails when the gateway silently changes labels without marking the route decision.