Signal magazine AI in Action Official Microsoft Blog Command Line Microsoft On The Issues Asia Canada Europe, Middle East and Africa Latin America The Code of Us What's new AI Innovation Digital Transformation Sustainability Security Work & Life Diversity & Inclusion Unlocked Microsoft 365 Azure Copilot Windows Surface XBOX Deals Small Business Support Windows Apps Outlook OneDrive Microsoft Teams OneNote Microsoft Edge Moving from Skype to Teams Computers Shop XBOX Accessories VR & mixed reality Certified Refurbished Trade-in for cash XBOX Game Pass Ultimate PC Game Pass XBOX games PC games Microsoft AI Microsoft Security Dynamics 365 Microsoft 365 for business Microsoft Power Platform Windows 365 Small Business Digital Sovereignty Azure Microsoft Developer Microsoft Learn Support for AI marketplace apps Microsoft Tech Community Microsoft Marketplace Software companies Visual Studio Microsoft Rewards Free downloads & security Education Gift cards Licensing Unlocked stories View Sitemap

By builders, for builders.

A Microsoft publication

Iteratively improving public benchmarks

Audit of coding benchmarks: DeepSWE and Terminal-Bench 4.0.

Independent Review: Hanwen Xing; SWE Review and PR Submissions: Daphne Barretto, Jennifer Marsman, Nishant Manchanda

Public benchmarks have become a common yardstick for measuring model capabilities, with much attention focused on scores and rankings. Yet through our careful examination of coding benchmarks like DeepSWE and Terminal-Bench 4.0, we found that the story behind those numbers is far more nuanced. Tasks can be ambiguous, graders can penalize valid solutions, and execution environments can introduce failures.

However, in a rapidly evolving AI landscape, no benchmark is likely to be perfect, nor should perfection be a prerequisite for learning from it. The real value of auditing public benchmarks lies not just in flagging defects, but in bringing continuous, rigorous judgment to the evaluation process, maintaining reliable capability coverage, learning from the findings, and improving them together as community efforts.

Below, we share our iterative auditing and learning process. Beyond identifying issues, we’ve also submitted fixes where no existing PRs were available and joined regular syncs with public benchmark maintainers to share findings and processes and collaborate on improvements. We hope to encourage everyone who uses public benchmarks to contribute to the community effort to continuously improve them.

Below is the learning funnel of us running public benchmarks:

When public benchmarks actually get it wrong

Public benchmark defect patterns

We carefully examined failure runs and identified benchmark-quality issues in 12 DeepSWE tasks and 10 Terminal-Bench 4.0 (TB4) tasks.

The findings revealed several recurring patterns, including task ambiguity, evaluability problems, grader-design issues, and evaluation-environment failures.

Audit findings by root cause

Root causeDeepSWETB4
Grader-design issues64
Task ambiguity & evaluability54
Evaluation-environment issues12
Total flagged1210

Deep dive on findings

Audit the audit

We believe benchmark auditing requires the right level of rigor and capability judgment. While defects should be iteratively identified and documented, flagging a defect does not necessarily mean a task should be filtered out or immediately fixed, as doing so could introduce selection bias and other potential issues. Through our iteration, we assess potential defects based on their likelihood of manifesting in practice and their material impact on capability measurement. This allows us to distinguish meaningful evaluation risks from theoretical concerns, preserving valuable benchmark coverage without compromising evaluation reliability.

Latent defects

In our audit, we identified latent defects—issues in task specifications or grading logic that exist but did not manifest in our observed runs. How to treat those tasks? Excluding every task with a known defect may seem like the safest approach, but it risks overcorrecting. Aggressive filtering can reduce capability coverage and introduce selection bias, particularly if more complex tasks are disproportionately removed.

Instead, we assess both the likelihood of a defect being triggered and its potential impact on scoring. We fix or filter when the risk is significant. Otherwise, we retain the tasks and continue monitoring them.

False pass

Beyond contamination and benchmark gaming, false passes can occur when an implementation passes all tests but violates a stated requirement, often because the tests miss relevant inputs or fail to check certain behaviors.

It is difficult to anticipate every edge case where an implementation can go wrong, as tests often reflect the failure modes the authors anticipated. Our principle is to take action when a missed failure reveals a meaningful model or harness capability gap. Otherwise, we document narrower limitations rather than trying to eliminate every minor imperfection.

Two adversarial test cases illustrate this distinction:

Evaluation reliability

Before attributing failures to benchmarks or model capabilities, we also need to ensure the evaluation itself is reliable. This includes reproducibility across runs and the stability of data pipelines that capture trajectories, transcripts, verifier logs, and execution artifacts. In our audit, issues with verifier-log ingestion and artifact persistence sometimes limited our ability to reconstruct runs and confidently attribute failures.

Then, it’s a real capability gap

Once we have ruled out benchmark defects and ensured the reliability of evals, we can focus on understanding the real capability gaps.

An important distinction is whether a failure comes from the model or the harness. Agent performance depends not only on the model’s reasoning and problem-solving abilities, but also on how the harness provides context, tools, and execution support. Identifying the root cause helps us make target improvements.

The following two examples illustrate how similar benchmark failures can reveal very different opportunities for improvement.

Looking ahead: Continuous improvement

Running and auditing public benchmarks has taught us that meaningful evaluation requires more than producing scores or identifying defects. It takes judgment, rigor, and a continuous cycle of learning and iteration.

We advocate for a more collaborative approach to public benchmarks—sharing our learnings with the community and working together to continuously improve them. Better benchmarks, in turn, help us set meaningful improvement goals, rigorously identify real capability gaps, and hill-climb toward better performance.

At Microsoft, we’re also developing internal benchmarks to better align capability measurements with product objectives and real people’s needs. We look forward to sharing more of those learnings in the future.