Respond to a cost-cap suspension¶

Your app stopped serving because accrued cost reached its cap. This guide diagnoses the breach and walks the two owner resume paths.

Preconditions¶

Owner (or platform) access to the App CR in tend-system.

Diagnose¶

1. Read the cost status¶

kubectl get app expense-helper -n tend-system \
  -o jsonpath='{.status.costStatus}' | jq
# state: "suspended"; compare accrued against spec.costCap.amount

2. Read the suspension record and condition¶

kubectl get app expense-helper -n tend-system \
  -o jsonpath='{.status.suspension}' | jq
# {"reason":"CostCapBreached","at":"…",
#  "resumableBy":"owner raises spec.costCap.amount above accrued cost,
#                 or applies the tendtech.io/resume annotation"}

kubectl get app expense-helper -n tend-system \
  -o jsonpath='{.status.conditions[?(@.type=="Suspended")]}' | jq

3. Check the event trail and the workloads¶

kubectl -n tend-system get events \
  --field-selector involvedObject.name=expense-helper
# CostWarning75, CostWarning90, CostSuspended — each fired once per crossing

kubectl get pods -n tend-app-expense-helper
# no running pods: suspend paused the app's Argo CD sync, then scaled to zero

You were warned twice before enforcement: warnings fire once per crossing of 75% and 90% (oscillation around a threshold does not re-fire them; the guards reset at period rollover).

Resume¶

Path A — raise the cap (the durable resume)¶

Set spec.costCap.amount above the accrued figure:

kubectl patch app expense-helper -n tend-system --type merge \
  -p '{"spec":{"costCap":{"amount":"100.00","currency":"EUR","period":"monthly"}}}'

Path B — the resume annotation¶

kubectl annotate app expense-helper -n tend-system tendtech.io/resume="true"

The operator consumes (removes) the annotation and lifts the suspension. If accrued cost is still at or over the cap, the next cost poll suspends again — the annotation is an explicit override for "I know, resume anyway"; raising the cap is the durable path.

What happens on resume¶

The operator restores the app's Argo CD sync and does not hand-scale workloads: GitOps self-heal restores the replicas from the desired state. On the GitOps-less local path there is no Argo CD to restore them, so workloads stay at zero — that is contract behavior, not a bug; scale them back up yourself there.

Verify it worked¶

kubectl get app expense-helper -n tend-system \
  -o jsonpath='{.status.costStatus.state}'
# normal (or warning-75/warning-90 if accrued is still above a threshold)

kubectl get app expense-helper -n tend-system \
  -o jsonpath='{.status.costStatus.lastResumedAt}'    # stamped on resume
kubectl get pods -n tend-app-expense-helper           # pods back (via GitOps)

status.suspension is cleared, the Suspended condition goes false, and a CostResumed event is recorded. The suspend and resume both land in the app's evidence record (Query an app's evidence).

Edge case: `state: unknown`¶

unknown means a telemetry gap in either cost source — OpenCost, or the model gateway's LLM-spend figure (status.costStatus.llmSpendState tells you which). Historically it meant only the first (OpenCost unreachable or returning nothing usable) — it is never enforced on. The app keeps running and the prior posture is left untouched until telemetry recovers. If you see unknown, fix cost telemetry (make verify-opencost NS=tend-app-expense-helper); do not expect a suspension or a resume while it lasts.