How we built ThinkingBox to measure whether agents finish the job

You ask a travel support agent to add a quiet-room preference to your hotel booking, away from elevators and ice machines. The agent collects your booking reference, checks the reservation, and says the request has been added. The transcript looks like a success. 

But the booking’s special_requests field is still empty. The agent confirmed work it never did. A grader looking only at the final response might pass the run. You discover the failure at check-in. 

We built ThinkingBox to catch this kind of failure. It runs agents in isolated, stateful tool environments, lets a simulated user answer follow-up questions, and inspects the side effects left behind. We’re also releasing ThinkingBox-Bench, a dataset of 507 executable tasks across five business domains. We evaluated 12 proprietary and open-weight models with 20 trials per task, then analyzed the traces to find where agents fail during real tool workflows.

What an agent benchmark needs to reproduce

Final-state evaluation works only when the benchmark controls the starting point and every way the state can change. ThinkingBox gives each task four connected pieces: 

1. A known initial state and expected outcome. The quiet-room task starts with an empty special_requests list and an open support ticket. It also defines the expected end state: the preference appears on the booking, and the ticket is solved with the correct resolution. The evaluator compares the actual records left by the agent with that expected outcome.
2. A controlled tool surface. The agent can change records only through the tools exposed for the task. Tool responses become its observations, while the evaluator retains a separate view of the underlying state.
3. A responsive user. The opening request doesn’t contain every required value. The simulated user provides the booking reference and room preference only when asked, so the benchmark can test whether the agent gathers the information needed for the correct update.
4. A clean environment for every attempt. Each run starts from the same baseline. Previous attempts can’t leave records or side effects that change the next result.

Together, these pieces let the benchmark attribute the final state to one agent run. The quiet-room task passes only if the preference appears on the booking and the support ticket records the correct resolution. The transcript and tool calls help explain how the agent got there, but neither can substitute for those updates.

Each ThinkingBox task therefore includes initial records, a user goal, available tools and policies, and executable assertions over the records left behind. The transcript explains the result. The assertions decide whether the task passes. 

In compact form, the quiet-room test looks like this:

The task defines the expected database state: the booking contains the quiet-room request and the support ticket is solved. The hash comparison checks the complete result against that state. The judge checks only the best-effort disclosure, which can be phrased in several valid ways.

Test outcomes, not trajectories

An executable environment still needs the right assertions. A test that prescribes the agent’s path can reject a valid solution:

An agent might inspect the support ticket first, skip a redundant profile lookup, or retry a read after a transient error. Any of those paths could complete the task. Tool order measures conformity to one reference trajectory. 

ThinkingBox checks the records left behind:

These assertions accept any trajectory that produces the required outcome. Tool calls still matter when a task requires or prohibits a specific action, but they are supporting evidence rather than the usual definition of success. 

Some requirements have no clean database value. The agent may need to tell the customer that a room preference is subject to availability. ThinkingBox handles this with a narrow judge question like, “Does the response state that the room preference is not guaranteed?” Deterministic assertions cover actions and records. Model-based grading covers meaning that can be expressed in several valid ways.

How ThinkingBox works

ThinkingBox separates the execution framework from the benchmark package. The thinkingbox repository contains the tb CLI, agent and simulated-user loop, MCP Session Proxy, and evaluation harness. The thinkingbox-data repository contains the MCP servers and synthetic records. Its scenario files select the servers and tools for a domain and provide shared instructions. Test cases add the user request, private user context, task-specific records, and checks. This split lets builders update the harness and benchmark independently. 

When you call tb infer, a task moves through six steps: 

1. Load the task. The CLI resolves the agent, scenario, and test case. It merges the scenario’s base records with task-specific data and selects the tools for the run.
2. Create an isolated session. The CLI asks the Session Proxy to create a session. The proxy starts the required MCP server processes and calls each server’s __reserved__init tool with the initial records.
3. Run the conversation. The agent receives the task query and calls tools through the proxy. The agent and simulated user use separately configurable LLM endpoints, so builders can change the model under evaluation without changing the task. The user model answers questions using private user_context; for the quiet-room task, that context contains your booking reference and preference details.
4. Collect the evidence. At the end of the conversation, ThinkingBox builds a TestContext with the final response, transcript, tool calls, and effects returned by each server’s __reserved__geteffects tool.
5. Execute the test. Python assertions evaluate the resulting records. A test can also ask a judge model a narrow semantic question when exact string matching would reject valid responses.
6. Destroy the session. The proxy calls __reserved__teardown and stops the MCP workers, even when the run fails.

tb infer can repeat each task and run attempts concurrently with a configurable batch size. Every attempt receives its own session ID and freshly initialized MCP processes, even when several attempts run in parallel. The long-running proxy routes their calls independently and tears down each session as it finishes. This keeps one failed or mutated run from affecting another while letting the benchmark use available model and tool capacity in parallel.

How the Session Proxy keeps runs isolated 

The Session Proxy makes parallel isolation possible. MCP standardizes individual tool calls; the proxy owns the lifecycle of each run. It invokes three reserved MCP tools for initialization, effect collection, and teardown. The agent never sees them. 

At the start of a run, tb infer sends the proxy a session ID, initial records, and a list of permitted tools. The proxy starts one process per MCP server, rather than one per tool. The travel benchmark packages its booking, support, CRM, and payment tools in one domain server, so each run starts one process that exposes those toolsets. A scenario configured with three servers would start three processes. Repeating the task creates fresh processes from the same initial records. 

The agent loop calls tools through MCPProxyClient. The client sends each request to the proxy over HTTP, and the proxy forwards it to the correct MCP process over stdio. The model sees ordinary tool definitions and results. HTTP is only the control boundary inside ThinkingBox. 

The agent loop and evaluator need different views of the same session. The agent loop calls only the permitted tools through the proxy. When the conversation ends, the harness uses the same session ID to retrieve effects and pass them to the test as TestContext; test fixtures can also inspect the live session when needed. A separate service adds a network hop, but it gives both paths one session boundary without exposing evaluator state to the agent. 

ThinkingBox-Bench: 507 executable business tasks

ThinkingBox provides the reusable machinery for stateful evaluation. In partnership with Toloka, we built ThinkingBox-Bench: 507 realistic, multi-turn tasks across five business domains. Each task requires an agent to take actions in complex policy environments. The comparison below uses the four requirements defined earlier.

Benchmark	Stateful environment	Controlled tool surface	Interactive user	Isolated, repeatable runs
SWE-bench	✅	❌	❌	✅
BFCL v3	✅	✅	❌	✅
ToolBench	❌	✅	❌	❌
API-Bank	❌	✅	❌	✅
WebArena	✅	❌	❌	✅
OSWorld	✅	❌	❌	✅
AppWorld	✅	✅	❌	✅
MCP-Atlas	❌	✅	❌	❌
tau-bench	✅	✅	✅	✅
tau2-bench	✅	✅	✅	✅
ThinkingBox-Bench	✅	✅	✅	✅

Most benchmarks cover only part of this evaluation contract. The tau-bench family is the closest comparison to ThinkingBox-Bench in that they also provide all four properties. ThinkingBox adds a reusable lifecycle around MCP servers and 507 tasks across five business domains, including auto insurance and internal IT and HR. 

The release manifest divides the 507 tasks across five fictional organizations: 

• Retail and e-commerce (98 tasks): Delivery exceptions, returns, refunds, exchanges, warranty claims, installations, promotions, and membership changes
• Travel and hospitality (104 tasks): Individual and group booking changes, payment recovery, cancellations, invoices, hotel verification, special requests, and post-stay complaints
• Auto insurance (100 tasks): Billing arrangements, proof-of-insurance documents, driver and vehicle changes, claim intake, reinstatement, and cancellation
• Neobank internal IT (104 tasks): Employee support workflows that combine identity, access, devices, software, and incident records
• Consulting IT and HR (101 tasks): Onboarding and offboarding, access provisioning, asset management, and employee-service workflows

What the leaderboard reveals

We ran every model on every task 20 times, then calculated pass@1 across all trials and tasks. Scores range from 65.36% for GPT-5.4 to 4.66% for Mistral-Large-3:

Model	Size	Retail	Auto insurance	Travel	Neobank	Consulting	Average
Proprietary models
GPT-5.4	—	76.33	62.65	68.13	65.34	54.60	65.36
GPT-5.2	—	70.20	22.40	53.70	51.15	34.06	46.28
o3-pro	—	37.94	2.96	24.16	24.37	14.75	20.60
Claude Sonnet 4.6	—	68.93	58.20	60.38	53.99	51.14	58.45
Claude Opus 4.6	—	74.90	14.65	28.89	38.03	34.21	37.91
Grok-4.3	—	43.93	2.60	15.14	1.78	9.55	14.38
Open-weight models
DeepSeek-V4-Pro	1.6T/49B	68.21	29.65	43.13	44.86	31.04	43.26
GLM-5.1	744B/40B	58.67	25.70	35.43	13.27	34.06	33.19
Kimi-K2.6	1T/32B	53.72	24.50	39.52	33.65	37.33	37.66
Mistral-Large-3	675B/41B	11.28	1.30	8.99	1.15	0.74	4.66
Qwen3.6-27B	27B	43.11	29.00	46.39	27.84	18.37	32.94
Qwen3.5-9B	9B	19.15	0.45	4.52	1.06	2.34	5.41

GPT-5.4 is the only evaluated model above 50% in every domain. Claude Sonnet 4.6 varies less by domain, while GPT-5.2 drops from 70.20% on retail to 22.40% on auto insurance. DeepSeek-V4-Pro, the strongest open-weight model in these results, approaches GPT-5.2 overall but has a different domain profile. The gap between the two Qwen models is also large, and some larger models rank below Qwen3.6-27B. Model size and a single aggregate score don’t tell you where an agent will succeed. 

The leaderboard separates systems, but it compresses 20 attempts per task into one average. That hides whether a model succeeds consistently or gets lucky.

Reliability is different from capability

Pass@1 asks how often a model finishes the task when run once; 20 trials let us ask two more questions: 

• pass@20: On what percentage of tasks did at least one of 20 attempts succeed?
• pass^20: On what percentage of tasks did all 20 attempts succeed?

Pass@20 measures whether retries can find a successful trajectory. Pass^20 measures whether the model can repeat it. 

GPT-5.4 leads the benchmark with 65.36% pass@1. Its pass@20 reaches 91.12%, so it found a successful trajectory at least once on most tasks. Its pass^20 is only 25.25%. It never passed 45 of the 507 tasks and passed all 20 attempts on 128. 

Similar pass@1 scores can hide very different profiles. Claude Opus 4.6 and Kimi-K2.6 score 37.91% and 37.66%. Opus reaches 70.02% pass@20 and 13.81% pass^20. Kimi reaches 84.22% pass@20 but only 3.16% pass^20. Kimi finds a solution on more tasks; Opus repeats successful behavior on a smaller set. 

Retries raise the chance of getting one good result without making the agent dependable. A builder deciding where to add guardrails, human review, or workflow constraints needs to know the difference. 

Diagnosing failed agent runs 

The assertions tell us whether the agent completed the task. To understand a failure, we inspect the conversation, tool calls, tool responses, final answer, and where the run stopped. We then assign each failed run to one main category. 

Failure category	Average share of failed traces	What appears in the trace
Tool usage	77.5%	A tool error, failed precondition, or unsuccessful lookup followed by no effective recovery
Wrong state change	12.1%	A mutating tool succeeds, but the agent applies the wrong entity, policy, value, or side effect
Response quality	7.9%	The agent gives an incomplete, contradictory, or premature resolution
Missing state change	2.5%	The agent performs lookups but never attempts the required mutation

Most agents get far enough to attempt the workflow, then fail during execution. Tool errors and unsuccessful lookups account for 77.5% of failed traces. Another 14.6% either stop before the required update or make the wrong update. Only 7.9% fail because of the final response. 

This tells us that agent loops need to interpret every tool result, change the plan after a failed action, and verify state before reporting success. ThinkingBox exposes that boundary: the trace records what the agent tried, and the effects show what changed. 

Try ThinkingBox 

ThinkingBox lives in two repositories. microsoft/thinkingbox contains the runtime. microsoft/thinkingbox-data contains the benchmark tasks, synthetic records, and MCP servers. 

To reproduce the benchmark, follow the ThinkingBox-Bench v1.0 release instructions. They cover installation, the 507-task evaluation run, and computing pass@1, pass@20, and pass^20 with tb agg. 

To build custom tools, scenarios, or tests, start with the ThinkingBox README and its end-to-end tutorial. The tutorial walks through creating an MCP server, defining a scenario, running an agent, and evaluating the result with executable assertions.