Skip to content

ADR-0019: Bifrost as the in-boundary model gateway

Status: Accepted — 004-gemini-inference (2026-07); amended 2026-08-12 (T21 groundwork — see "Amendment" below) Principles served: I, II, IV; Platform & Security Baseline (recorded deviation) Source: specs/004-gemini-inference/research.md R1, R5, T001 spike outcomes; specs/004-gemini-inference/plan.md Complexity Tracking; specs/004-gemini-inference/contracts/model-gateway-contract.md. Amendment: Bifrost docs/features/observability/default.mdx (branch dev), plugins/telemetry/main.go at the pinned transports/v1.6.3, and the transports/v1.6.10 release notes; docs/tasks.md T21.

Context

Real Tier-1 inference (Gemini on Vertex AI, feature 004) makes per-app LLM budgets load-bearing, and they must be enforced inside the boundary, because GCP cannot cap Gemini spend (verified 2026-07-12): current Gemini serving uses dynamic shared quota, so there is no per-project rate or token knob to lower; Cloud Billing budgets alert but never stop spend; and the only documented hard stop — budget alert → Pub/Sub → function → detach billing — kills every resource in the project and Google warns data may be irretrievably deleted. That kill switch was rejected as destructive collateral for a governance platform. So the architecture gains a model gateway slot between the inference gateway and cloud providers, owning provider adaptation, cloud credential refresh, per-app virtual keys, and hard budgets. No CNCF component provides per-consumer LLM budgets, spend attribution, and cloud-provider credential refresh in OSS.

Decision

Bifrost (Maxim AI, Apache-2.0) occupies the slot: self-hosted in tend-system, pinned at transports/v1.6.3 with the image digest recorded and the image mirrored into Harbor, delivered by GitOps as an Argo CD application (deploy/model-gateway/). Persistence is a SQLite config store on a PVC — providers, virtual keys, budgets, and usage counters, which is wanted state — while the request-logging store is explicitly disabled so no prompt or response content is persisted anywhere (logs_store.enabled: false plus client.enableLogging: false and disableContentLogging: true; merely omitting the section default-recreates the store). Bifrost's OSS tier includes virtual keys, hierarchical budgets, and per-consumer cost tracking (REST + Prometheus), and authenticates to GCP with a Workload Identity Federation external_account config with automatic token refresh. Bedrock and Azure OpenAI support pre-pays the multi-cloud future.

Alternatives considered

  • Envoy AI Gateway — architecturally ideal (CRDs, WIF-capable backend security policy, no content logging by design, fully Apache-2.0), but it requires Kubernetes ≥ 1.32 while the dev cluster pins 1.31.9, brings the full Envoy Gateway stack, and emits token metrics only — no dollar cost and no budget enforcement.
  • LiteLLM — richest day-one spend features, but a Python service with a mandatory Postgres, and JWT/OIDC auth is enterprise-gated.
  • Kong AI Gateway — token cost features paywalled.
  • APISIX — service-account-key-only GCP auth.
  • Portkey — acquired by Palo Alto (2026-06); "OSS 2.0" unreleased.
  • Helicone — maintenance mode post-acquisition.
  • TensorZero — no per-consumer governance; noted as a possible future prompt-experimentation layer, a distinct concern. Archived 2026-06-12: repository read-only, project wound down by its founders. The seam this bullet left open therefore has no named candidate — see "Amendment".
  • Extending Tend's own inference gateway — reimplements provider adaptation, OAuth/WIF token refresh, token metering, and budget accounting: undifferentiated heavy lifting and a permanent maintenance surface.

The baseline deviation, and its mitigation

Bifrost is not a CNCF component, so this deviates from ADR-0001's baseline. The deviation is justified and recorded where ADR-0001 requires: the plan's Complexity Tracking (specs/004-gemini-inference/plan.md). The mitigation is that Bifrost is replaceable plumbing behind the inference-gateway contract: nothing app-visible, builder-visible, or App-CRD-visible depends on Bifrost specifically (FR-013). The narrow surface any replacement must satisfy is specs/004-gemini-inference/contracts/model-gateway-contract.md — an OpenAI-compatible data plane with per-app virtual keys and pre-flight budget refusal, key/budget administration and spend-read APIs from which the operator can rebuild all state from App CRDs alone, and zero content persistence. Swapping means providing that same surface, repointing the inference gateway's upstream URL and the operator's admin/spend client, rebuilding keys from App CRDs, and re-running the quickstart — no app, builder, App CRD, or policy contract changes.

Consequences

In-boundary per-app budgets become the primary spend control for model usage; GCP billing alerts at 50/90/100% are the backstop only, and in-boundary spend figures are governance numbers, not invoice numbers — the model-gateway pricing decision says so, and the Tier-1 Posture Record records the provider's own billing alerts as the invoice-truth backstop. The platform carries a recorded non-CNCF dependency, pinned by version and digest, so upgrades are deliberate acts against a young (v2.x-era) project. Zero content persistence rests on explicit configuration rather than defaults, so the quickstart verifies it. Envoy AI Gateway is re-evaluated against the same contract when a cluster bump reaches Kubernetes ≥ 1.32.

Amendment (2026-08-12) — four corrections, none reversing the decision

Raised while scoping T21 (builder authoring quality), which needs a cost signal per authoring run and asked what this ADR left open.

1. The prompt-experimentation seam has no candidate. TensorZero, the one named above, was archived 2026-06-12. The field was surveyed against Tend's constraints and the conclusion is no dedicated experimentation layer — recorded with its reasoning in docs/tasks.md T21. The short form: every platform in that class (Langfuse, Opik, Phoenix, MLflow) earns its keep by being the store of prompts, traces and outcomes, and 010 already decided that store is the app's own Gitea repo with "no second store to keep in step". Adopting one would also stand up five or six stateful services and put every tenant's prompts in a new in-boundary component to mirror, attest and defend. This ADR's seam stays open and unfilled, deliberately.

2. Metadata-only request logging is available, and this ADR's own wording invited over-reading. "Zero content persistence" was implemented as two belts — no logs store AND no logging at all — which is stricter than the contract requires. Bifrost's built-in observability is self-contained (no external service; SQLite or Postgres; native dashboard), and disable_content_logging: true with the logs store enabled keeps per-request usage metadata while storing no request or response bodies. That middle setting is compatible with contract #8 as written. Adopting it is a deliberate act, not a default: it creates a per-request table carrying app attribution, and values.yaml documents a chart gotcha where the binary default-creates logs.db — so any move here is gated on verifying live that the rows carry no prompt text. Nothing has been changed yet.

3. What the OSS tier has is not what this deployment exposes. The Decision above says the OSS tier includes per-consumer cost tracking "(REST + Prometheus)". True of the product; false of the deployment. The telemetry plugin — which is what emits Prometheus metrics — is plugins.telemetry.enabled: false, and the rendered config.json registers only the governance plugin. So neither channel is live, and the governance virtual-key counters are the only spend surface today. Read at the pinned tag, plugins/telemetry/main.go emits input/output/cache tokens and bifrost_cost_total in USD, labelled by provider, model, virtual_key_id/_name, key_id/_name, path, method, status and cache_type — no request content in any label, and cost computed by the pricing manager, which here reads the git-versioned local datasheets. Enabling it is the cheap, content-free way to get the cost dimension.

4. Bifrost's own evaluation and prompt-management story is out of bounds. It arrives as a connector to Maxim AI's hosted platform — Bifrost's own vendor's SaaS — which would send prompt and response content off-cluster and is refused under Principle I, on the same grounds as re-enabling content logging in the one pod with off-cluster egress. The connectors that fit are OpenTelemetry and Prometheus. This does not weaken the choice of Bifrost; the gateway slot never wanted those features.

Version drift, noted while checking the above. Pinned here at transports/v1.6.3; upstream is at transports/v1.6.10 (2026-08-12), actively maintained. It adds per-model budgets and rate limits per virtual key and quarterly budget windows — both directly useful to the operator's per-app budget path, and the per-model budget is what an experiment arm wants. It also ships a non-reversible database migration, so that upgrade is a deliberate act needing a config.db backup, exactly as the Consequences above anticipate.