Krevo AI diagnosed 14 of 14 controlled Kubernetes failures.
The dated production-path run achieved 100% strict and semantic accuracy on the covered corpus. This is a bounded, reproducible benchmark—not a claim that every Kubernetes incident will be diagnosed correctly.
Published result
The verdict matched the scenario’s expected failure class, used compatible evidence, and presented a remediation direction appropriate to that controlled fault.
Method
- Deploy one labeled failure scenario into the benchmark cluster.
- Wait for the KrevoPilot agent to report the affected resource and evidence.
- Run the investigation through the normal production product path.
- Capture verdict, probable cause, evidence and proposed fix.
- Compare the output with the scenario’s ground-truth label and semantic acceptance criteria.
- Reset the scenario before continuing.
The published run uses the 14 controlled failure scenarios listed below. The report records the exact expected class, observed class and result for every scenario.
Covered failure classes
| Area | Controlled classes exercised | Result |
|---|---|---|
| Images and startup | Image pull failure, wrong command, init-container crash | Passed |
| Configuration | Missing ConfigMap, missing Secret, missing ServiceAccount | Passed |
| Health checks | Wrong readiness port, liveness kill loop | Passed |
| Scheduling and capacity | Node-selector mismatch, insufficient resources, unbound PVC | Passed |
| Runtime and network | CPU throttling, DNS failure, NetworkPolicy egress block | Passed |
Scenario names, expected classes and evidence assertions are version-controlled so a regression can fail CI instead of silently changing product behavior.
Strict and semantic scoring
The primary verdict maps to the exact expected failure class. A contradictory primary diagnosis fails even if another section mentions the right cause.
The explanation identifies the operational cause and points to a safe, relevant remediation even when wording differs from the label.
A useful result must remain internally coherent: object path, failing value, evidence, hypothesis state and fix cannot contradict the verdict.
Limitations
Real incidents can combine multiple faults, involve managed services outside the cluster, depend on missing telemetry, or present novel failure modes.
- The corpus is controlled and intentionally labeled.
- Accuracy depends on collected events, status, manifests, metrics and available logs.
- Privacy redaction can remove exact values needed for a specific patch.
- LLM-backed reasoning can vary; deterministic guards and evidence checks reduce but do not remove risk.
- Generated commands and YAML must be reviewed by an authorized operator.
- The result is dated; every release should publish a new run rather than reuse this number indefinitely.
Verification and publication
KrevoPilot runs the controlled benchmark in an isolated test cluster, never against production workloads. Every supported failure class has a ground-truth label, required evidence and expected safe action. Release checks reject accuracy regressions and contradictory fixes.
Each published run records the application revision, agent version, chart version, cluster version, included scenarios, exclusions, strict score, semantic score and known limitations.