Research

Sphinx sets a new state-of-the-art on SREGym

We set a new state of the art on an SRE benchmark using in-context learning.

Date

Nov 15, 2026

Author

Harshal Chamdal, Rohan Kodialam

The AI SRE is an increasingly popular agent that takes ownership of diagnosing and fixing problems in production software. As AI accelerates the pace of software creation, maintaining that software is becoming an increasingly critical priority — startups like Traversal and Resolve AI that build AI SREs are seeing growing enterprise adoption, and general-purpose coding agents like Cognition’s Devin increasingly benchmark on SRE tasks. Incumbents in the observability space like Datadog are also augmenting their platforms with agentic capabilities.

There are two complementary approaches to building the best AI SRE. First, post-training on SRE work to create a better base model. Second, post-training on a specific environment to teach an agent the nuances of working in a certain tech stack and platform.

Recently, Datadog released Scout 1.0, a model trained specifically for production issues using simulations from their own incidents and alert data. This fine-tuned model performed quite well on benchmarks like SREGym, showing the merits of the first approach.

However, we believe that the second approach is even more productive to building powerful, customized intelligence for AI SREs. Rather than focusing on specializing individual models, Sphinx focuses on specializing context. Every team’s production systems, alerts, and procedures are unique, and change as time passes. Sphinx studies a team’s setup, practices, and how they’ve evolved to train a graph of context that is injected into prompts at inference time to influence and improve agent behavior.  

To validate this method we trained Sphinx on production repositories, incident reports, post-mortems and synthetic agent traces from SREGym. The value of this training is that it allows for generalization to new production issues. To prevent leakage, Sphinx never saw the SREGym faults (tasks), the grader, or the answers during training. 

With a Sphinx context graph, GPT-6.1 Sol (Codex Harness) and Opus 5.5 (Claude Code Harness) both resolve more end-to-end SREGym tasks than any previously published agent  - including Datadog’s specialized Scout model. On the full set of 90 faults:

  • GPT-6.1 Sol post-trained with Sphinx resolves 86 issues, vs 82 for GPT-6.1 Sol in Codex, and 80 for Scout 1.0 (as per the Scout Technical Report)

  • Claude 5.5 Haiku post-trained with Sphinx can resolve 80 issues, matching the performance of Scout 1.0. Sphinx enables out-of-the-box small models to perform as well as specialized models like Scout.

Faults resolved end to end, out of 90
Published run Scout 1.0 With Sphinx (ours)
0153045607590Sphinx sets a new state-of-the-art on SREGym - Sphinx86GPT-6.1 Sol + SphinxSphinx sets a new state-of-the-art on SREGym - Sphinx84Claude Opus 5.5 + SphinxSphinx sets a new state-of-the-art on SREGym - Sphinx83GPT-6 Astra + SphinxSphinx sets a new state-of-the-art on SREGym - Sphinx82GPT-6.1 SolSphinx sets a new state-of-the-art on SREGym - Sphinx80Claude Haiku 5.5 + SphinxSphinx sets a new state-of-the-art on SREGym - Sphinx80Scout 1.0Sphinx sets a new state-of-the-art on SREGym - Sphinx80GPT-6 AstraSphinx sets a new state-of-the-art on SREGym - Sphinx79Claude Opus 5.5Sphinx sets a new state-of-the-art on SREGym - Sphinx76Claude Fable 5.1Sphinx sets a new state-of-the-art on SREGym - Sphinx68GLM-5.3Sphinx sets a new state-of-the-art on SREGym - Sphinx66GLM-5.3-FlashSphinx sets a new state-of-the-art on SREGym - Sphinx65GPT-5.6 Sol
Published scores: Datadog’s Scout 1.0 report, trimmed to current models. Sphinx scores: our runs.

Sphinx generalizes across models

Fine-tuning brought GLM-5.3 Flash from 66 faults to Scout's 80, up to the frontier. However, Sphinx moves the frontier itself, not just one model. The same in-context learning created a lift across every model we ran, from GPT-6 Astra down to Claude Haiku 5.5 and GPT-6 Luna.

Notably, gains are larger for smaller models like Haiku and Luna. Since these models are generally weaker, we expect post-training (including in-context learning) to have a more significant uplift.

Sphinx moves every model up
No SphinxWith Sphinx
75% 80% 85% 90% 95% $0.02 $0.05 $0.1 $0.2 $0.5 $1 $2 $5 Cost per fault (log scale)Faults resolved end to end Sphinx sets a new state-of-the-art on SREGym - Sphinx 76.7% Sphinx sets a new state-of-the-art on SREGym - Sphinx GPT-6 Luna84.4% Sphinx sets a new state-of-the-art on SREGym - Sphinx 80.0% Sphinx sets a new state-of-the-art on SREGym - Sphinx Haiku 5.588.9% Sphinx sets a new state-of-the-art on SREGym - Sphinx 91.1% Sphinx sets a new state-of-the-art on SREGym - Sphinx GPT-6.1 Sol95.6% Sphinx sets a new state-of-the-art on SREGym - Sphinx 87.8% Sphinx sets a new state-of-the-art on SREGym - Sphinx Opus 5.593.3% Sphinx sets a new state-of-the-art on SREGym - Sphinx 88.9% Sphinx sets a new state-of-the-art on SREGym - Sphinx GPT-6 Astra92.2%
Without Sphinx, Sol, Opus and Astra start from Datadog's published scores, placed at our own runs' cost since Datadog reports none. Haiku and Luna start from our runs.

Our setup for SREGym

We evaluate using SREGym, a public benchmark, where agents run in a live Kubernetes cluster with one injected fault. The agent must first diagnose the fault, then fix it – a task succeeds end-to-end only if the system is healthy afterward, which a grader checks by confirming that the relevant alerts have cleared.

We primarily focus on the full benchmark (90 tasks, no noise) in our evaluation. For our rollouts, each model is run using its provider’s corresponding harness; that is, Claude Code for Haiku and Opus models and Codex for Luna, Sol, and Astra models. We allow only one attempt per fault, similar to Datadog’s reporting. We run our live clusters using compute from Modal.

Sphinx trained on the information that a human SRE team would have collected for their apps – their deployments and Helm charts, configuration, source code, alert rules, and the team's practices for making and checking changes – building a single context graph that was used across all runs. To prevent leakage, none of the SREGym faults or grader logic was used for training. Agents were limited in access during runs, with access only to the live cluster they are debugging, SREGym’s API to allow agents to submit solutions, and context injected from Sphinx.

Why studying ahead helps

Frontier models are already good at finding the root cause of an issue in SREBench. However, what seems to differentiate specialized systems is creating a correct fix: changing a live system the way the team does, and confirming the change worked. With Sphinx enabled, agents are more able to precisely undo changes that cause issues, and to check the signals the team uses before claiming the investigation and resolution are completed.

Example: A misconfigured rolling update. 

In this example, a change affected two parts of a deployment: 1) it added an init container that never finishes; 2) it set an update strategy that takes every replica down at once.

Without Sphinx, half the bad change stays
Sphinx page readLast changeSubmit
01234567No SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxNot resolvedSphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxResolvedSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxFixes the init container,leaves the strategyFixes bothWaits for alerts to clearMinutes

Running in Claude Code with no additional training, Haiku 5.5 removes the init container and leaves the strategy "as a follow-up". With Sphinx, the same model reads the team's page on making changes and then it reverts both fields and waits for the alerts to clear.

Example: Payments Flag

The fix worked, but the alert hadn't cleared
Sphinx page readLast changeSubmit
0123456No SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxNot resolvedSphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxResolvedSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSphinx sets a new state-of-the-art on SREGym - SphinxSubmits 44 s after the fixWaits for three alert-free minutesMinutes

A feature flag was set to make every payment fail. Without Sphinx, GPT-6.1 Sol found the flag, turned it off and restarted the flag service. Five test checkouts went through, and it submitted ~45 seconds after the fix with the error-rate alert still firing. 

With Sphinx, it made the same fix but also watched Prometheus and submitted only after three minutes without an alert. In both cases, the agent made the correct fix but only with Sphinx did one check the actual signal the team uses.

Access to context isn’t enough, but training with Sphinx closes the gap.

Code and config files describe a system’s current state, but taking action on a live system requires understanding of how the team operates. If agents could pick this up by reading system files and logs, a trained context graph wouldn't be needed – we find that this is not the case 

Our agents work “closed book”: they see only the live cluster. We also ran them open book, with the apps' Helm charts, configuration and docs on disk for agents to grep or search them. As an ablation, we also gave access to the raw data used to train Sphinx without the actual context graph. 

We found that flooding agents with raw context actually slightly decreased performance. On the other hand, using that same information as training data for Sphinx dramatically improved performance. 

An open book adds nothing; Sphinx adds 7.8 points
Points gained over no Sphinx, average of five models
Sphinx sets a new state-of-the-art on SREGym - Sphinx-0.7Open bookSphinx sets a new state-of-the-art on SREGym - Sphinx+7.8SphinxSphinx sets a new state-of-the-art on SREGym - Sphinx+3.6Sphinx + open book

 A brief note on SREGym Lite

SREGym Lite is a 17-fault subset of SREGym with a popular public leaderboard. This sub-benchmark is saturated, with several systems correctly solving every fault. That makes it hard for us to measure improvements on just this subset of tasks – however for completeness we find that GPT-6 Astra + Sphinx resolves all 17 faults, matching the performance of other leading systems. 

System

Model

Effort

Resolved

Sphinx + Codex

GPT-6 Astra

high

100%

Devin Oncall

Claude Opus 5.5

high

100%

Devin Oncall

GPT-6 Astra

max

100%

CloudThinker

GPT-6 Astra

low

100%

CloudThinker

GPT-6 Astra

max

100%

Other entries are from the SREGym Lite leaderboard. Devin Oncall averaged two attempts a fault; the others ran three.

Future work

Stacked Gains from Post-training and Context Graphs

Datadog’s Scout specializes a model while Sphinx specializes context. Each moves SREGym on its own, but we have yet to determine if the gains are cumulative if both methods are used together. A direction we’re exploring is running Scout with a Sphinx knowledge base on SREGym so we can better understand where the two methods overlap and where there are synergies we can exploit to further boost performance. 

Evaluating Sphinx on Continual Learning 

A post-trained model is trained at a single point in time, but a team’s systems keep changing after that. Services get added, alerts get tuned, and every incident teaches the on-call team something new. It is computationally and operationally expensive to continually re-train after each change – however, we explicitly designed Sphinx to evolve easily, which means it can keep learning after deployment. Each incident and each agent rollout becomes evidence that confirms or corrects what the context graph already captures. SREGym lets us demonstrate strong performance on a “static” system, and we’re working to develop evals that measure how well Sphinx does at continual learning in a realistic dynamic environment.