Research
Sphinx sets a new state-of-the-art on SREGym


We set a new state of the art on an SRE benchmark using in-context learning.
Date
Nov 15, 2026
Author
Harshal Chamdal, Rohan Kodialam
The AI SRE is an increasingly popular agent that takes ownership of diagnosing and fixing problems in production software. As AI accelerates the pace of software creation, maintaining that software is becoming an increasingly critical priority — startups like Traversal and Resolve AI that build AI SREs are seeing growing enterprise adoption, and general-purpose coding agents like Cognition’s Devin increasingly benchmark on SRE tasks. Incumbents in the observability space like Datadog are also augmenting their platforms with agentic capabilities.
There are two complementary approaches to building the best AI SRE. First, post-training on SRE work to create a better base model. Second, post-training on a specific environment to teach an agent the nuances of working in a certain tech stack and platform.
Recently, Datadog released Scout 1.0, a model trained specifically for production issues using simulations from their own incidents and alert data. This fine-tuned model performed quite well on benchmarks like SREGym, showing the merits of the first approach.
However, we believe that the second approach is even more productive to building powerful, customized intelligence for AI SREs. Rather than focusing on specializing individual models, Sphinx focuses on specializing context. Every team’s production systems, alerts, and procedures are unique, and change as time passes. Sphinx studies a team’s setup, practices, and how they’ve evolved to train a graph of context that is injected into prompts at inference time to influence and improve agent behavior.
To validate this method we trained Sphinx on production repositories, incident reports, post-mortems and synthetic agent traces from SREGym. The value of this training is that it allows for generalization to new production issues. To prevent leakage, Sphinx never saw the SREGym faults (tasks), the grader, or the answers during training.
With a Sphinx context graph, GPT-6.1 Sol (Codex Harness) and Opus 5.5 (Claude Code Harness) both resolve more end-to-end SREGym tasks than any previously published agent - including Datadog’s specialized Scout model. On the full set of 90 faults:
GPT-6.1 Sol post-trained with Sphinx resolves 86 issues, vs 82 for GPT-6.1 Sol in Codex, and 80 for Scout 1.0 (as per the Scout Technical Report)
Claude 5.5 Haiku post-trained with Sphinx can resolve 80 issues, matching the performance of Scout 1.0. Sphinx enables out-of-the-box small models to perform as well as specialized models like Scout.
Sphinx generalizes across models
Fine-tuning brought GLM-5.3 Flash from 66 faults to Scout's 80, up to the frontier. However, Sphinx moves the frontier itself, not just one model. The same in-context learning created a lift across every model we ran, from GPT-6 Astra down to Claude Haiku 5.5 and GPT-6 Luna.
Notably, gains are larger for smaller models like Haiku and Luna. Since these models are generally weaker, we expect post-training (including in-context learning) to have a more significant uplift.
Our setup for SREGym
We evaluate using SREGym, a public benchmark, where agents run in a live Kubernetes cluster with one injected fault. The agent must first diagnose the fault, then fix it – a task succeeds end-to-end only if the system is healthy afterward, which a grader checks by confirming that the relevant alerts have cleared.
We primarily focus on the full benchmark (90 tasks, no noise) in our evaluation. For our rollouts, each model is run using its provider’s corresponding harness; that is, Claude Code for Haiku and Opus models and Codex for Luna, Sol, and Astra models. We allow only one attempt per fault, similar to Datadog’s reporting. We run our live clusters using compute from Modal.
Sphinx trained on the information that a human SRE team would have collected for their apps – their deployments and Helm charts, configuration, source code, alert rules, and the team's practices for making and checking changes – building a single context graph that was used across all runs. To prevent leakage, none of the SREGym faults or grader logic was used for training. Agents were limited in access during runs, with access only to the live cluster they are debugging, SREGym’s API to allow agents to submit solutions, and context injected from Sphinx.
Why studying ahead helps
Frontier models are already good at finding the root cause of an issue in SREBench. However, what seems to differentiate specialized systems is creating a correct fix: changing a live system the way the team does, and confirming the change worked. With Sphinx enabled, agents are more able to precisely undo changes that cause issues, and to check the signals the team uses before claiming the investigation and resolution are completed.
Example: A misconfigured rolling update.
In this example, a change affected two parts of a deployment: 1) it added an init container that never finishes; 2) it set an update strategy that takes every replica down at once.
Running in Claude Code with no additional training, Haiku 5.5 removes the init container and leaves the strategy "as a follow-up". With Sphinx, the same model reads the team's page on making changes and then it reverts both fields and waits for the alerts to clear.
Example: Payments Flag
A feature flag was set to make every payment fail. Without Sphinx, GPT-6.1 Sol found the flag, turned it off and restarted the flag service. Five test checkouts went through, and it submitted ~45 seconds after the fix with the error-rate alert still firing.
With Sphinx, it made the same fix but also watched Prometheus and submitted only after three minutes without an alert. In both cases, the agent made the correct fix but only with Sphinx did one check the actual signal the team uses.
Access to context isn’t enough, but training with Sphinx closes the gap.
Code and config files describe a system’s current state, but taking action on a live system requires understanding of how the team operates. If agents could pick this up by reading system files and logs, a trained context graph wouldn't be needed – we find that this is not the case
Our agents work “closed book”: they see only the live cluster. We also ran them open book, with the apps' Helm charts, configuration and docs on disk for agents to grep or search them. As an ablation, we also gave access to the raw data used to train Sphinx without the actual context graph.
We found that flooding agents with raw context actually slightly decreased performance. On the other hand, using that same information as training data for Sphinx dramatically improved performance.
A brief note on SREGym Lite
SREGym Lite is a 17-fault subset of SREGym with a popular public leaderboard. This sub-benchmark is saturated, with several systems correctly solving every fault. That makes it hard for us to measure improvements on just this subset of tasks – however for completeness we find that GPT-6 Astra + Sphinx resolves all 17 faults, matching the performance of other leading systems.
System | Model | Effort | Resolved |
Sphinx + Codex | GPT-6 Astra | high | 100% |
Devin Oncall | Claude Opus 5.5 | high | 100% |
Devin Oncall | GPT-6 Astra | max | 100% |
CloudThinker | GPT-6 Astra | low | 100% |
CloudThinker | GPT-6 Astra | max | 100% |
Other entries are from the SREGym Lite leaderboard. Devin Oncall averaged two attempts a fault; the others ran three.
Future work
Stacked Gains from Post-training and Context Graphs
Datadog’s Scout specializes a model while Sphinx specializes context. Each moves SREGym on its own, but we have yet to determine if the gains are cumulative if both methods are used together. A direction we’re exploring is running Scout with a Sphinx knowledge base on SREGym so we can better understand where the two methods overlap and where there are synergies we can exploit to further boost performance.
Evaluating Sphinx on Continual Learning
A post-trained model is trained at a single point in time, but a team’s systems keep changing after that. Services get added, alerts get tuned, and every incident teaches the on-call team something new. It is computationally and operationally expensive to continually re-train after each change – however, we explicitly designed Sphinx to evolve easily, which means it can keep learning after deployment. Each incident and each agent rollout becomes evidence that confirms or corrects what the context graph already captures. SREGym lets us demonstrate strong performance on a “static” system, and we’re working to develop evals that measure how well Sphinx does at continual learning in a realistic dynamic environment.