Governing an AI-proposed fix before it reaches Kubernetes

The problem: a plausible fix is not enough

A Deployment starts crashing after a rollout. An investigator proposes a rollback. Before applying it, an operator still needs to know whether the rollout brought the failure, what the rollback would undo, whether GitOps owns the workload, and whether the live object is still the one that was investigated.

I built Themis around those decisions. An Alertmanager alert opens an incident; the core collects evidence and evaluates checks. A model, an external agent or an engineer can propose a fix, but every proposal goes through the same governance path.

The architecture follows permission boundaries

Core · Go

Reads cluster evidence, evaluates CEL checks and policy, serves the incident UI, records approvals, and coordinates verification. PostgreSQL holds state, audit records and the River job queue.

Investigator · Python

Runs a bounded investigation over evidence obtained from the core through gRPC. It can propose a fix but holds no Kubernetes service-account or database credentials.

Executor · Go

The only component with permission to patch workloads. It is optional and disabled by default; when enabled, its write role is limited to Deployments in enrolled namespaces.

The decisions that matter

• Derive the change from live state. The core constructs the action from Kubernetes objects. The operator sees what changes, rather than relying on the model's description of what it intends.
• Bind approval to the action. A digest covers the target identity and planned change. The executor recomputes it against live state, performs a server-side dry run and writes with an optimistic-concurrency precondition. An uncertain write outcome is recorded as ambiguous rather than retried blindly.
• Keep investigation separate from justification. A proposal needs a true justifying check and must pass policy. Logs can inform a diagnosis; text in evidence cannot authorize a fix by itself. Argo CD and Flux ownership leads to a recommendation instead of a competing write.
• Check the symptom after the write. Verification measures the outcome over a window. A harmful fix can trigger a governed, automatic restoration attempt; if it cannot be restored, Themis escalates with the change still in place.

What the alpha demonstrates

The first release covers five incident classes on Deployments: bad rollouts, low memory limits, CPU throttling, probe restart storms and stuck pods. Fixes include full or partial rollback, resource changes and probe timing changes; stuck-pod cases are referred to a person. The release includes a Helm chart and images for amd64 and arm64.

The evaluation contains 91 synthetic cases, including 28 prompt-injection variants. In the reported qwen3 8B run, the investigator attempted 14 harmful fixes and governance refused all 14. CI replays recorded model replies as a regression gate. The kind end-to-end suite uses a fake model to exercise approval, refusal, execution, verification and recovery.

These are bounded test results. They do not establish safety on unseen incidents or prove production reliability. The alpha supports Deployments and Alertmanager only; restoration is best effort, and configuration and APIs may change.