Inference gateway¶

Complete reference for the inference gateway — the only path for model inference in Tend (Principle I). It exposes an OpenAI-compatible endpoint, authenticates the builder, and proxies to the tier-configured upstream model endpoint; prompts, responses, and telemetry stay in-boundary. For why the tiers exist see boundary tiers; for the model/prompt provenance it feeds see the evidence chain.

Sources of truth: services/inference-gateway/ and deploy/inference-gateway/.

Endpoints¶

Endpoint	Behavior
`POST /v1/chat/completions`	OpenAI-compatible chat completion, proxied to the configured upstream. Requires builder auth (below).
`GET /healthz`	Liveness; always `200 ok`.
`GET /readyz`	Readiness; `200 ok` when the upstream URL and OIDC issuer are configured, `503` otherwise — traffic is refused before the gateway is usable.

Request/response semantics¶

The request body is proxied verbatim except the model field, which is always overwritten with the configured GATEWAY_MODEL. The builder cannot choose an off-boundary or unconfigured model — this is the enforcement point for Principle I. The gateway does not inspect prompt content; it records only metadata.
The upstream response is returned verbatim (status and body), so the resolved model in the response body reaches the builder adapter for provenance recording.
Response headers set by the gateway:

Header	Value
`X-Tend-Model`	The resolved model id (always the configured model, authoritative).
`X-Tend-Tier`	The configured boundary tier (`tier1`/`tier2`), for audit.
`X-Request-ID`	Caller-supplied `X-Request-ID`, or a generated id, for in-boundary tracing.

Error responses:

Status	When
`401 unauthorized`	Missing/malformed `Authorization` header, bad signature, wrong issuer or audience, expired token, or unreachable JWKS — all fail closed to 401.
`400 bad request body`	Body exceeds `GATEWAY_MAX_REQUEST_BODY_BYTES` or cannot be read.
`502 upstream error`	The upstream request could not be built or failed.

Builder authentication¶

Two token classes are accepted, both Keycloak-issued: the builder's client-credentials token (identified by azp matching GATEWAY_BUILDER_CLIENT_ID, default tend-builder), and a per-app token carrying a non-empty tendtech.io/app claim, which the gateway maps to that app's own virtual key so spend and provenance attribute per app. A validated token that is neither class is refused 401 with no upstream call.

The builder adapter authenticates with a Keycloak client-credentials OIDC access token (the tend-builder confidential client), sent as Authorization: Bearer <jwt>. On every request the gateway validates:

Check	Detail
Signature	RS256 only, against keys from a cached JWKS document. The cache is fetched lazily; an unknown `kid` triggers a refresh, rate-limited to one per minute; the JWKS HTTP client has a 10 s timeout.
Issuer	`iss` must equal `OIDC_ISSUER` exactly.
Audience	`aud` must contain `OIDC_AUDIENCE` (default `tend-inference-gateway`).
Expiry	`exp` is required and validated (plus `nbf` when present).

The authenticated builder subject — the token's azp (authorized party) claim when present, else sub — flows into telemetry and provenance.

There is no static-token fallback (FR-021). The walking skeleton's shared service token is gone; any validation failure is a 401. A permanent fallback would hollow out the control.

Configuration¶

All configuration is environment variables; boundary-sensitive parameters (upstream URL, tokens) are never embedded in the binary or image. The gateway refuses to start when a required variable is missing or GATEWAY_TIER is invalid.

Variable	Required	Default	Meaning
`GATEWAY_LISTEN_ADDR`	no	`:8080`	HTTP listen address.
`GATEWAY_UPSTREAM_URL`	yes	—	Tier 1 model endpoint URL (e.g. an Azure OpenAI deployment).
`GATEWAY_UPSTREAM_TOKEN`	yes	—	API key / bearer token for the upstream. Injected at runtime from a Secret; never logged (startup logs the upstream URL, not the token).
`OIDC_ISSUER`	yes	—	Keycloak realm issuer URL, e.g. `http://keycloak.tend-system.svc/realms/tend`.
`OIDC_AUDIENCE`	yes	`tend-inference-gateway`	Required `aud` claim on builder tokens.
`OIDC_JWKS_URL`	no	derived: `<issuer>/protocol/openid-connect/certs`	JWKS endpoint override (exists mainly for tests).
`GATEWAY_MODEL`	yes	—	Resolved model id/version, e.g. `anthropic.claude-3-5-sonnet-20241022`. Returned in every response for provenance.
`GATEWAY_TIER`	no	`tier1`	`tier1` \| `tier2`; any other value refuses startup.
`GATEWAY_BUILDER_CLIENT_ID`	no	`tend-builder`	The `azp` value that identifies a token as builder-class.
`GATEWAY_VK_DIR`	no	`/etc/tend/virtual-keys`	Mount of the operator-managed per-app virtual-key Secret, one file per app. Empty disables per-app virtual-key resolution.
`GATEWAY_REQUEST_TIMEOUT`	no	`120s`	Maximum wait for the upstream (sized for long-form code generation).
`GATEWAY_MAX_REQUEST_BODY_BYTES`	no	`4194304` (4 MiB)	Maximum request body size.

Tier behavior¶

The boundary tier is contract, not code: the binary does not branch on GATEWAY_TIER. Which boundary the gateway enforces is entirely a matter of what GATEWAY_UPSTREAM_URL points at:

Tier	Upstream	Status
`tier1`	Managed model endpoint in the customer's own cloud tenancy and region, reached over public-internet TLS rather than private interconnect	The default; the tier this slice runs.
`tier2`	Fully self-hosted serving on customer-operated infrastructure (in-cluster)	Reserved; configuration is a later feature.

Swapping the model, or moving between tiers, is a configuration change (Secret/env update), not a code change. The tier value is surfaced in the X-Tend-Tier header and telemetry for audit. The spec.tier field on the App CRD selects which boundary an app's builder runs against.

Deployment facts¶

From deploy/inference-gateway/ (delivered via Argo CD):

Property	Value
Namespace	`tend-system`
Deployment	`inference-gateway`, 1 replica, image `harbor.tend-system.svc/platform/inference-gateway:latest`
Service	`inference-gateway.tend-system.svc`, port `8080` (ClusterIP only — not reachable from outside the cluster)
ServiceAccount	`inference-gateway`, `automountServiceAccountToken: false`
Secret	`inference-gateway-config` with keys `upstream-url`, `upstream-token`, `model`. The committed values are working development defaults — the in-cluster model gateway, its seeded platform virtual key, and the pinned model. Config validation rejects only an empty value, so these pass and the gateway starts: an installation must replace them, not merely fill them in.
Non-secret env	`OIDC_ISSUER=http://keycloak.tend-system.svc/realms/tend`, `OIDC_AUDIENCE=tend-inference-gateway`, `GATEWAY_TIER=tier1` — plain env in the Deployment, not secrets.
Probes	Liveness `GET /healthz`, readiness `GET /readyz`.
Security context	`runAsNonRoot`, seccomp `RuntimeDefault`, no privilege escalation, all capabilities dropped.
Telemetry	Request metadata logged to stdout, in-cluster. No telemetry, prompts, or responses leave the configured boundary tier.

Cilium network policy (ADR-0007) enforces that no other pod may reach the upstream endpoint directly — the gateway is the only egress path to the model.