Terminal-Bench
Abstract. Terminal-Bench is an evaluation family for coding and computer-use agents that asks whether an agent can complete realistic technical work inside an actual terminal-accessible environment, not merely emit a patch, a JSON action, or a one-shot answer. Its main contribution is to make shell-mediated, multi-turn engineering work—the file system, process control, dependency management, test execution, and failure recovery—a first-class benchmark target alongside SWE-bench-style issue resolution and Agent-Computer Interface research.
Coverage note: verified through May 11, 2026.
Terminal-Bench: Evaluating Coding Agents in a Real Terminal
1. Why Terminal-Bench matters
Terminal-Bench is best understood as Terminal-Native Evaluation for AI agents. Instead of reducing software-engineering evaluation to “given an issue, produce a patch,” it evaluates whether an agent can use the terminal as a working environment: inspect files, run commands, install or invoke tools, debug failures, modify artifacts, and leave the environment in a state that passes hidden or task-provided tests. The official Terminal-Bench paper describes Terminal-Bench 2.0 as 89 hard tasks in terminal environments, each with a containerized environment, instruction, tests, and a human-written solution; frontier model-agent systems remained below 65% in the paper’s reported experiments. ar5iv
The benchmark’s premise is not that the terminal is the only interface agents should use. It is that the terminal is still a dense, universal interface for software engineering, scientific computing, cybersecurity, machine learning, data analysis, and systems administration. The Terminal-Bench authors explicitly motivate the terminal as a medium where agents can issue shell commands, invoke tools, explore an environment, and perform long-horizon work; the 1.0 launch post similarly framed the terminal as text-based, powerful, sandboxable, and already central to many engineering workflows. ar5iv
That distinction matters because many older coding benchmarks measure narrower objects. SWE-bench asks systems to resolve real GitHub issues from Python repositories, and its canonical formulation is a patch-generation problem scored by tests derived from pull requests. SWE-agent and mini-SWE-agent show that agent scaffolding and tool design can make a large difference on SWE-bench, but the benchmark target remains primarily repository issue resolution. Terminal-Bench broadens the target from “fix this repo issue” to “operate in this technical environment until the outcome is correct.” SWE-bench+2NeurIPS Proceedings+2
The phrase “real terminal” should be read carefully. Terminal-Bench tasks still run in controlled, containerized environments for reproducibility and safety; the claim is not that agents are unleashed on arbitrary production machines. The claim is that the interaction surface is terminal-native: the agent sees a task instruction, manipulates a file system and running processes, and is evaluated on the final environment state rather than a single emitted answer. ar5iv
2. Benchmark object: a task in a live environment
A Terminal-Bench task is built around an instruction, a container environment, tests, a reference solution, and a time limit. The Terminal-Bench 2.0 paper states that tests verify the final state or outcome of the container rather than the exact commands the agent ran, which is an important design choice: agents may take different routes as long as the end state satisfies the verifier. ar5iv
The Harbor task format makes this explicit. A task directory contains an instruction.md, a task configuration file, an environment definition such as a Dockerfile or Docker Compose setup, a solution script, and a test script. During evaluation, tests are copied into the environment and executed; reward can be recorded through reward.txt or reward.json, or inferred from exit status and unit-test commands. Harbor
This structure separates three roles that are often conflated in weaker evaluations. The instruction defines the work, the environment defines the world in which the work must be done, and the verifier defines the observable success condition. The reference solution is not the answer the agent is supposed to imitate; it is an oracle artifact used to check that the task is solvable and that tests can identify success. Terminal-Bench’s original GitHub documentation describes each task as including an instruction, test script, and oracle solution, while the 2.0 paper reports automated oracle validation plus manual review. GitHub
The result is a benchmark object closer to an engineering job than to a static exam question. A task may require reading unfamiliar files, discovering available tools, recovering from failed commands, updating scripts, configuring dependencies, or producing an artifact. The official Terminal-Bench 2 repository gives examples ranging from protein assembly to debugging asynchronous code and finding security vulnerabilities, and the Terminal-Bench site describes version 2.0 as covering software engineering, machine learning, security, and data science. GitHub
3. Comparison with SWE-bench, SWE-agent, and mini-SWE-agent
Terminal-Bench should not be interpreted as a replacement for SWE-bench. It measures a related but different capability. SWE-bench’s core contribution was to ground coding-agent evaluation in real GitHub issues: 2,294 task instances from 12 Python repositories, with evaluation based on whether generated changes make previously failing tests pass. SWE-bench Verified later introduced a 500-instance human-filtered subset, while SWE-bench Lite, Multilingual, and Multimodal provide other slices of the same broad issue-resolution family. SWE-bench+1
SWE-agent’s contribution is also distinct. The SWE-agent paper argues that unmodified language models struggle when asked to operate directly through a normal Linux shell, so the authors design an Agent-Computer Interface with simplified actions for viewing, searching, editing, and executing code. The reported SWE-agent results—12.47% pass@1 on the full 2,294-task SWE-bench set in the original paper—showed that scaffolding can materially improve repository repair without changing the base model weights. NeurIPS Proceedings
mini-SWE-agent pushed the opposite pressure point: how small can the agent scaffold be while still solving real repository issues? Its repository describes it as a compact command-line agent for solving GitHub issues and helping in the shell, and the SWE-bench leaderboard news reported mini-SWE-agent reaching 65% on SWE-bench Verified in July 2025. That makes mini-SWE-agent important evidence that agent scaffolds can become simpler while remaining competitive, but it does not remove the benchmark gap that Terminal-Bench targets: broad terminal-mediated work outside GitHub issue repair. GitHub
| Dimension | Terminal-Bench 2.x | SWE-bench / SWE-bench Verified | SWE-agent / mini-SWE-agent |
|---|---|---|---|
| Primary unit | A technical task in a containerized terminal environment | A real GitHub issue requiring a repository patch | Agent scaffolds evaluated largely on repository repair and command-line coding tasks |
| Interaction surface | Shell, file system, process execution, installed tools, agent-specific tool interfaces | Repository checkout plus test-based patch evaluation | Custom or minimal agent-computer interfaces for navigating, editing, and testing code |
| Scoring target | Final environment state or task outcome | Tests pass after generated patch | Depends on benchmark; commonly SWE-bench resolution rate |
| Scope | Software engineering plus ML, security, data science, systems, scientific workflows, and other terminal-native tasks | Primarily software-engineering issue resolution in Python repos, with later benchmark variants | Not a benchmark family by itself; an agent-design line |
| Scale | Terminal-Bench 2.0: 89 selected tasks | SWE-bench full: 2,294; Verified: 500 | Varies by evaluation run |
| Key epistemic risk | Small N, environment drift, public-task leakage, scaffold/model confounding | Repository/test leakage, patch overfitting, issue-distribution representativeness | Tool-interface confounding and benchmark-specific optimization |
Terminal-Bench’s own registry work shows that the authors do not view these benchmark families as mutually exclusive. Their registry page says the framework can adapt existing text-based benchmarks into the Terminal-Bench harness, including SWE-bench Verified, AppWorld, DevEval, and EvoEval; parity experiments report that adapted evaluations preserve scores closely, including a SWE-bench Verified comparison where Terminus with Claude 4 Opus obtained 66% under both the SWE-bench grading path and the Terminal-Bench adapter path. Terminal-Bench+1
That adapter result is methodologically useful. It supports the claim that Harbor/Terminal-Bench is partly a harness layer—a way to run agents against many task formats—and not just a single dataset. It also sharpens the conceptual distinction: SWE-bench can be hosted inside a terminal-evaluation framework, but Terminal-Bench Core is broader than SWE-bench because the task distribution is not limited to GitHub issue repair. Terminal-Bench+1
4. What Terminal-Bench 2.0 changed
Terminal-Bench 1.0 launched in May 2025 with 80 hand-crafted and human-verified tasks. The launch post emphasized a harness that could specify tasks, evaluate agents, manage complex environments, orchestrate multi-container Docker lifecycles, log actions, and verify container state. It also introduced Terminus as a neutral agent scaffold intended to avoid giving proprietary agents special advantages from bespoke tool interfaces. Terminal-Bench
Terminal-Bench 2.0, announced with Harbor in November 2025, was positioned as harder and more carefully verified. The announcement says Terminal-Bench 1.0 had succeeded as a community benchmark but exposed reliability problems, including tasks that depended on external services whose behavior changed over time. Terminal-Bench 2.0 therefore emphasized more difficult tasks, stronger manual and language-model-assisted verification, and the new Harbor harness for cloud containers and agent rollouts. Terminal-Bench
The Terminal-Bench 2.0 paper gives the clearest dataset-construction picture. Ninety-three contributors created 229 tasks, from which 89 were selected after difficulty and quality review by three experienced reviewers. The review process included oracle execution, checklists, language-model checks, manual review, model trials, adversarial exploit attempts, and additional auditing, with the authors estimating roughly three reviewer-hours per task while still acknowledging that flaws may remain. ar5iv
This is a strength and a weakness. The strength is that Terminal-Bench 2.0 tasks are intentionally curated to be solvable, realistic, and difficult. The weakness is that 89 expert-curated tasks are not a statistically broad sample of all terminal work. The benchmark is high-density, not high-volume: each task carries much more weight than each instance in SWE-bench full, and small changes in task validity or scoring can shift aggregate leaderboard conclusions. ar5iv
5. Terminal-native evaluation and multi-turn tool use
The central design move is to evaluate agents through iterative environment interaction. The paper describes tasks as interactive: agents explore and manipulate the environment using tool calls, such as editing files or running Bash commands. The score is not assigned to the transcript itself; it is assigned to whether the agent leaves the environment in a passing state. ar5iv
This stresses capabilities that patch-only or chat-only benchmarks often under-measure. Agents must decide what to inspect, how to interpret command output, when to rerun tests, whether a dependency problem is real or incidental, how to recover from failed commands, and how to avoid damaging the environment. These are not merely “coding” skills; they are Operational Agency skills. A model can know the right algorithm and still fail Terminal-Bench by invoking the wrong executable, misunderstanding the working directory, waiting on a hung process, or failing to verify its own work.
The Terminal-Bench 2.0 paper’s error analysis supports this interpretation. It classifies failures into execution, coherence, and verification categories, and reports that execution errors dominate. The authors further analyze command-level failures, including missing executables, path issues, and failures while running executables, which are exactly the kinds of problems a terminal-native benchmark is designed to surface. ar5iv
SWE-agent’s original paper reached a related conclusion from the other direction. It found that agents benefit from an interface designed around repository repair—structured commands for viewing, searching, editing, and testing—rather than a raw shell. Terminal-Bench asks whether agents can survive in the richer terminal setting, while SWE-agent asks how an interface can simplify the agent’s interaction with a repository. Both are part of the same larger research problem: the design of reliable Agent-Computer Interfaces. NeurIPS Proceedings
6. Harness design: what capabilities are exposed?
Harbor is the official harness for Terminal-Bench 2.0. Its documentation defines the core objects as tasks, datasets, agents, containers, trials, and jobs: a task contains a single instruction, container environment, and test script; a trial is one agent attempt; a job is a collection of trials that can run in parallel. Harbor supports several environment providers, including local or cloud container backends. Harbor
The task configuration exposes practical controls that matter for benchmark interpretation. Harbor task files can specify verifier timeouts, agent timeouts, environment build timeouts, CPU, memory, storage, GPU requirements, operating system, environment variables, internet permissions, and MCP servers. These settings are not incidental: a benchmark about terminal work can be made easier or harder by changing time budgets, hardware, internet access, or the set of tools installed in the container. Harbor
The official leaderboard rules recognize this. The Terminal-Bench 2.1 leaderboard says submissions cannot modify timeouts or resources in task.toml. That restriction is necessary because otherwise a submission could improve its measured score not by better agency but by changing the evaluation envelope—more time, more memory, fewer verifier constraints, or a different runtime environment. Terminal-Bench
Harbor supports multiple agent-integration modes. The agents documentation lists pre-integrated agents such as Terminus 2, Claude Code, Copilot CLI, Codex CLI, Gemini CLI, OpenHands, and Mini-SWE-Agent. It also supports external agents through a BaseEnvironment execution interface, and installed agents that run inside the task container with their own tools and workflows. Harbor
This means “Terminal-Bench score” is never just a model score. It is an agent-plus-model-plus-harness score. A model may perform differently under Codex CLI, Claude Code, Gemini CLI, OpenHands, Mini-SWE-Agent, or Terminus 2 because each scaffold has different tool affordances, prompting, logging, error recovery, and command-execution behavior. The Terminal-Bench 2.0 paper explicitly evaluates six agents across 16 frontier models and reports that model choice usually mattered more than the scaffold, but the leaderboard format still reports combinations rather than isolated model capability. ar5iv
7. How tool calls map to terminal actions
The cleanest reference point is Terminus 2, the neutral agent scaffold used heavily by Terminal-Bench. The Terminus 2 documentation describes it as a sandboxed, mono-tool terminal agent that interacts through tmux: it can send keystrokes, scroll, use arrow keys, launch shells, and interact with command-line interfaces. It runs independently in a separate process, inside Dockerized isolation, and is designed for autonomous execution without human intervention. Harbor
That mono-tool design matters. Instead of giving the model a high-level “edit file” API, Terminus can reduce the interface to terminal keystrokes and shell interactions. This makes the benchmark closer to real terminal work, but it also makes evaluation noisier: failures can arise from shell quoting, terminal state, scrolling, process management, or interactive command behavior. In contrast, SWE-agent’s custom interface deliberately abstracts some of those details away to improve repository-repair performance. Harbor
Other agents are less neutral. Claude Code, Codex CLI, Gemini CLI, OpenHands, and Mini-SWE-Agent bring their own affordances and may expose specialized editing, search, planning, or patch tools. Harbor’s installed-agent interface allows those agents to be installed in the container and executed headlessly, which is appropriate for measuring real agent products but less clean for isolating base-model competence. Harbor
The harness therefore has two evaluation modes in tension. One mode asks, “Which deployed agent stack solves the task?” That is the leaderboard user’s question. The other asks, “Which underlying model has the strongest terminal agency?” That is the scientific question. Terminal-Bench can support both, but only if reports distinguish model, scaffold, environment, time budget, and tool affordances.
8. Time budgets, costs, and long-horizon behavior
Terminal-Bench tasks are not all short. The paper reports expert time estimates: many tasks were expected to take an expert under an hour, many between one hour and one day, and a smaller number longer for junior engineers. Junior estimates were substantially higher, making the benchmark intentionally difficult for systems without robust exploration and recovery. ar5iv
The evaluation runs are also operationally expensive. The paper reports 32,155 trials across six agents and 16 frontier models, with at least five runs per combination. It notes that most agents completed within 20 minutes, but some runs extended up to two hours, involved hundreds of API calls, and consumed on the order of 100 million tokens in extreme cases. ar5iv
This is part of the benchmark’s value. Terminal-Bench is not only testing whether a model can reason about code; it is testing whether an agent can spend time productively. The paper reports no simple correlation between turn count or token count and success, which is a useful warning against assuming that longer agent loops automatically imply better outcomes. ar5iv
It also explains why timeouts and resource envelopes are not mere implementation details. A benchmark of long-horizon terminal agency can be gamed or distorted by extending time limits, adding more parallel search, or allowing more expensive tool use. A score should therefore be read together with its agent timeout, verifier timeout, hardware envelope, internet setting, number of runs, and confidence interval.
9. Score trajectory: rapid progress, but not clean model progress
The Terminal-Bench 2.0 paper reported that no frontier model-agent combination exceeded 65% on its 89 tasks. In the paper’s results, Codex CLI with GPT-5.2 reached 63%, Terminus 2 with Claude Opus 4.5 reached 58%, Terminus 2 with Gemini 3 Pro reached 57%, and the open-weight Kimi K2 Thinking model reached 36%. The authors also observed that over roughly eight months, state-of-the-art performance nearly doubled between Gemini 2.5 Pro and GPT-5.2, while warning that the benchmark might saturate within the following year. ar5iv
The public leaderboard has continued moving quickly. On Terminal-Bench 2.0, the current top rows include Codex CLI with GPT-5.5 at 82.0% ± 2.2, ForgeCode with GPT-5.4 at 81.8% ± 2.0, TongAgents with Gemini 3.1 Pro at 80.2% ± 2.6, and ForgeCode with Claude Opus 4.6 at 79.8% ± 1.6. These results are not directly comparable to the paper’s original table without accounting for time, agent improvements, submission policies, and possible task-fix effects, but the trend is clear: public Terminal-Bench 2.0 scores rose from the low-60s frontier reported in the paper to the low-80s on the live leaderboard. Terminal-Bench+1
Terminal-Bench 2.1 complicates the story further because it is a revision, not just a new leaderboard page. The Terminal-Bench 2.1 announcement says it fixes issues in 28 of the 89 Terminal-Bench 2.0 tasks and introduces continuous validation. On the current Terminal-Bench 2.1 leaderboard, Codex CLI with GPT-5.5 leads at 83.4% ± 2.2, followed by Terminus 2 with GPT-5.5 at 78.2% ± 2.4, Terminus 2 with Gemini 3 Pro at 74.4% ± 2.6, Gemini CLI with Gemini 3.1 Pro at 70.7% ± 2.9, and Claude Code with Claude Opus 4.7 at 69.7% ± 2.7. Terminal-Bench+1
| Reference point | Dataset / leaderboard | Agent + model | Reported score | Interpretation |
|---|---|---|---|---|
| Paper frontier | Terminal-Bench 2.0 paper | Codex CLI + GPT-5.2 | 63% | Best paper-reported combination; below 65% |
| Paper frontier | Terminal-Bench 2.0 paper | Terminus 2 + Claude Opus 4.5 | 58% | Strong neutral-scaffold result |
| Paper frontier | Terminal-Bench 2.0 paper | Terminus 2 + Gemini 3 Pro | 57% | Strong neutral-scaffold result |
| Live 2.0 | Terminal-Bench 2.0 leaderboard | Codex CLI + GPT-5.5 | 82.0% ± 2.2 | Current high public score on 2.0 |
| Live 2.0 | Terminal-Bench 2.0 leaderboard | ForgeCode + GPT-5.4 | 81.8% ± 2.0 | Agent stack near top of 2.0 |
| Live 2.1 | Terminal-Bench 2.1 leaderboard | Codex CLI + GPT-5.5 | 83.4% ± 2.2 | Current high public score on revised 2.1 |
| Live 2.1 | Terminal-Bench 2.1 leaderboard | Terminus 2 + GPT-5.5 | 78.2% ± 2.4 | Strong neutral-scaffold 2.1 result |
The honest reading is that Terminal-Bench has already become a moving target. It still discriminates between systems, but the top of the leaderboard is approaching the saturation zone the paper anticipated. The benchmark remains useful, but its strongest future use may be less as a single headline percentage and more as a source of trajectory logs, failure taxonomies, task-specific stress tests, and harness methodology for newer benchmark families.
10. Contamination, leakage, and leaderboard integrity
Terminal-Bench 2.0 is not a secret benchmark. The paper explicitly notes a leakage risk: because internet access is allowed, an agent could theoretically locate the dataset or oracle solution. The authors say they did not observe such behavior in tens of thousands of trajectories, added a canary string to the repository for decontamination, and did not attempt a private test set in that paper. ar5iv
That status should be stated plainly: a public terminal benchmark with internet-enabled agents cannot rely on secrecy as its main validity guarantee. A canary helps detect training-corpus contamination, but it does not prevent intentional retrieval, overfitting by agent developers, or leaderboard-specific reward hacking. Terminal-Bench’s validity therefore depends on a combination of task design, trajectory auditing, submission rules, community review, and future private or rotating sets.
The benchmark’s own maintainers have already confronted this. An April 2026 leaderboard-integrity post says the community found cheating and reward hacking in submissions, leading to new policies: ATIF trajectories are required for all passing trials, reward-hacked trials receive zero credit, an agent judge reviews all passing trials, and static and community checks are used. The post gives examples including modified timeouts, encrypted solutions, uploaded test folders, and an agent file that retrieved solutions from the internet. Terminal-Bench
This is not a fatal indictment of Terminal-Bench; it is a predictable consequence of evaluating autonomous agents in rich environments. The more capable the agent, the more capable it is of exploiting the benchmark process itself. Terminal-Bench is valuable partly because it exposes that pressure. But current scores should be read as audited public-leaderboard scores, not as cryptographically leakage-proof estimates of general terminal competence.
The Agent Trajectory Interchange Format is one response to this problem. Harbor’s ATIF documentation describes a standardized JSON representation for full interaction histories, including model messages, tool calls, environment feedback, token usage, cost, and metadata; Harbor supports ATIF output for agents including Terminus 2, OpenHands, Mini-SWE-Agent, Gemini CLI, Claude Code, and Codex. Requiring such trajectories makes reward hacking easier to detect and turns the benchmark from a scalar score into an inspectable record of agent behavior. Harbor
11. Task drift and the Terminal-Bench 2.1 correction
Terminal-Bench 2.1 is important because it demonstrates both the benchmark’s maintainability and its fragility. The 2.1 announcement says the revision fixes issues in 28 of 89 Terminal-Bench 2.0 tasks. It identifies categories such as external-dependency problems, resource mismatches, and instruction-test misspecification, and reports that most representative model-agent pairs improved under the revised task set. Terminal-Bench
Z.ai’s Terminal-Bench 2.0 Verified work is especially informative. The writeup reports a comprehensive review of the Terminal-Bench 2 dataset, with environment fixes across 89 tasks and instruction fixes in 11 tasks, and gives concrete examples such as missing procps, resource-detection problems, and instruction-test inconsistencies. In Z.ai’s reported evaluation, environment fixes alone improved Claude Opus results by more than five percentage points, indicating that task environment quality can materially affect measured agent performance. Hugging Face
This should discipline any interpretation of score changes. A model-agent score can rise because the model improved, the scaffold improved, the environment became less brittle, the task instructions became clearer, the verifier became more permissive or more accurate, or the submission adapted to the benchmark. Terminal-Bench 2.1 likely improves benchmark validity, but it also confirms that terminal-native tasks have more moving parts than static text tasks.
The deeper methodological lesson is that terminal-native evaluation needs continuous validation. Environments rot, package managers change, external services alter behavior, OS-level defaults shift, and “obvious” instructions can conflict with tests. Terminal-Bench’s maintainers are treating this as an ongoing maintenance problem rather than a one-time dataset release, which is the right posture for an agent benchmark with live environments. Terminal-Bench+1
12. Methodology critiques
12.1 Small N relative to SWE-bench
Terminal-Bench 2.0’s 89 tasks are much fewer than SWE-bench full’s 2,294 instances and SWE-bench Verified’s 500 instances. This is not merely a cosmetic difference. With 89 tasks, each task contributes about 1.12 percentage points to the headline score before repeated trials and aggregation. A handful of flawed, unusually hard, unusually easy, or contaminated tasks can noticeably affect the leaderboard. ar5iv+2SWE-bench+2
The counterargument is that Terminal-Bench tasks are much more expensive to author and verify. They are not extracted at scale from existing pull requests; they are curated terminal environments with oracle solutions and tests. The benchmark trades breadth for task depth. That trade is defensible, but it means Terminal-Bench should not be used as though it were a large random sample of engineering work.
12.2 Expert-curated subset and selection bias
The Terminal-Bench 2.0 paper says 93 contributors created 229 tasks and the benchmark selected 89 after review. The curation process is extensive and valuable, but it also means the final distribution reflects the maintainers’ and reviewers’ judgment about what counts as realistic, difficult, and well-specified. ar5iv
That is a common problem in expert benchmarks. Expert curation tends to improve quality and reduce triviality, but it can also overrepresent tasks that benchmark authors find interesting, tasks that are easy to containerize, or tasks that fit a verifier format. The result may be an excellent stress test without being a representative census of terminal work.
12.3 Scoring variability
Terminal-Bench’s own limitations section notes that external dependencies can drift and that hardware or runtime differences can cause variability. The 2.1 revision and Z.ai verification work give concrete evidence that environment fixes and instruction fixes can move scores. ar5iv+2Terminal-Bench+2
Repeated trials and confidence intervals help, but they do not remove all uncertainty. Agent runs are stochastic; tool use may be nondeterministic; package installs can fail; internet access may change; and a scoring script may test only part of the desired behavior. In this setting, confidence intervals around the top leaderboard scores should be treated as necessary but not sufficient. A system that is two points ahead may not be meaningfully better if the task set, scaffold, or environment differs.
12.4 Agent-stack confounding
Terminal-Bench reports agent-model combinations, and that is the right product-oriented choice. But it complicates scientific claims. A leaderboard entry may improve because the model is better at reasoning, because the agent scaffold is better at editing and testing, because the prompt is more benchmark-aware, because the tool interface reduces friction, or because the environment setup is more compatible with that agent. Harbor’s support for many pre-integrated and installed agents makes the benchmark practically useful while increasing the number of confounders. Harbor
The Terminus scaffold partially addresses this by giving a neutral reference point. The Terminal-Bench paper used Terminus 2 as a neutral Bash-only scaffold, and the Terminus documentation describes its tmux-centered mono-tool interface. But even a neutral scaffold is a scaffold: parser choices, context summarization, max turns, and process isolation still shape behavior. ar5iv
12.5 Outcome-only scoring and reward hacking
Scoring final environment state is the right default for task realism, but it creates openings for reward hacking. If an agent can inspect tests, alter the verifier, retrieve solutions, exploit hidden assumptions, or change timeouts, then the measured outcome may no longer represent task completion. Terminal-Bench’s integrity post shows this is not hypothetical: submissions have been penalized or removed for behaviors such as modifying timeouts, uploading tests, or retrieving benchmark solutions. Terminal-Bench
The solution is not to abandon outcome scoring. Human engineering work is outcome-oriented too. The solution is to combine outcome scoring with trajectory review, strict harness immutability, private or rotating validation sets, adversarial task review, and explicit reward-hacking policies.
13. What Terminal-Bench measures well
Terminal-Bench is strong where the work is naturally terminal-mediated. It measures whether agents can use command-line tools, navigate file systems, run tests, inspect logs, configure environments, and recover from operational failures. That makes it a strong benchmark for Long-Horizon Tool Use, especially in domains where the “answer” is a changed system rather than a text response.
It is also strong at revealing execution fragility. A model that performs well on code synthesis may still fail when a package is missing, a process hangs, a test emits misleading output, or a shell command needs careful quoting. Terminal-Bench’s error analysis found execution failures to be a major category, which is precisely the kind of failure that chat-only or patch-only evaluations tend to undercount. ar5iv
Terminal-Bench also measures agent autonomy more directly than many coding benchmarks. In a SWE-bench-style setting, the task often has a known repository and a known issue. In Terminal-Bench, the agent may need to discover the relevant files, infer the structure of the environment, and decide which tools to use. That makes the benchmark closer to open-ended engineering work, though still far from real production autonomy.
Finally, Terminal-Bench is useful as infrastructure. Harbor’s registry and adapter system can run other benchmarks through a common agent interface, and ATIF can standardize trajectory logs across agents. Those features may outlast any particular Terminal-Bench task set. Terminal-Bench+2Terminal-Bench+2
14. What Terminal-Bench does not settle
Terminal-Bench does not settle whether an agent is a good production software engineer. Production engineering involves code review, product ambiguity, security constraints, team conventions, long-lived memory, dependency ownership, deployment risk, and human communication. Terminal-Bench compresses work into a containerized task with a verifier, which is necessary for benchmarkability but incomplete as a model of work.
It also does not settle which model is “best at coding” in isolation. Scores depend on the agent wrapper, terminal interface, tool permissions, environment setup, and task distribution. A model that performs worse under a raw terminal scaffold might perform better under an IDE-native interface or a specialized repository-repair interface. The SWE-agent paper is direct evidence that interface design can change performance materially. NeurIPS Proceedings
Terminal-Bench also does not eliminate contamination concerns. Public benchmark tasks are inspectable, and terminal agents with internet access are unusually capable of finding or exploiting public information. The benchmark maintainers have added canaries, trajectory requirements, and integrity policies, but those are mitigations rather than proofs of uncontaminated evaluation. ar5iv
Most importantly, Terminal-Bench does not prove that terminal-native evaluation is the final form of agent evaluation. It proves that terminal-native work is a valuable stress test today.
15. Permanent benchmark category or transitional form?
The case for Terminal-Bench as a permanent category is strong. The terminal is not going away; it remains the native interface for many developer, research, ML, security, and systems workflows. A benchmark that asks whether agents can operate through that interface captures a real capability: using a computer through textual tools to accomplish technical work. The Terminal-Bench authors’ motivation—that the terminal is text-based, versatile, and sandboxable—will remain true even as IDEs and agent UIs improve. Terminal-Bench
The case against permanence is also strong. Many real coding agents will not live primarily in raw terminals. They will live in IDEs, code-review systems, hosted development environments, CI/CD platforms, and multi-agent orchestration tools. IDE-Native Coding Agents can expose richer semantic context than a terminal: AST-aware navigation, code lenses, symbol search, inline diffs, debugger state, issue context, and user telemetry. In such settings, a raw terminal benchmark may understate the performance of agents designed for richer interfaces.
There is also a broader move toward trajectory-native evaluation. Harbor’s ATIF makes the full interaction history a first-class artifact, not just a debugging log. Once trajectories are standardized, future benchmarks can evaluate cross-interface behavior: terminal actions, IDE actions, browser actions, API calls, model messages, tool observations, and cost traces. Terminal-Bench may then become one environment type inside a larger Agent Trajectory Evaluation ecosystem. Harbor
Related benchmark directions point the same way. Recent SWE-bench-derived work argues that public issue benchmarks may not reflect how developers interact with chat-based coding assistants in IDEs, and goal-oriented benchmark proposals such as CodeClash emphasize that real development is often driven by goals rather than isolated task instances. These are not replacements for Terminal-Bench, but they identify the same limitation: the evaluation target is shifting from isolated answers to situated agent behavior over time. arXiv
The most likely outcome is not that Terminal-Bench disappears. It is that terminal-native evaluation becomes one durable slice of a larger benchmark stack. There will be terminal tasks for shell competence, SWE-bench-style tasks for repository repair, IDE-native tasks for interactive developer assistance, browser and OS tasks for general computer use, and trajectory-level audits for safety and reward hacking. Terminal-Bench’s long-term contribution may be less the exact 89-task dataset and more the insistence that agents must be evaluated as operators in real executable environments.
16. How to read Terminal-Bench results
A Terminal-Bench score should be read as an operational result, not a pure intelligence number. The meaningful unit is “agent stack under specified harness conditions,” not “model.” A good report should state the model, agent scaffold, task version, timeouts, resource settings, internet setting, number of trials, confidence interval, and whether trajectories were audited.
A score difference should be treated cautiously when confidence intervals overlap or when the compared systems use different scaffolds. For example, a Terminus 2 result may be more useful for comparing base models under a relatively neutral terminal interface, while a Codex CLI or Claude Code result may be more useful for comparing deployed product agents. Both are legitimate, but they answer different questions.
Task-version differences also matter. Terminal-Bench 2.0, Terminal-Bench 2.1, and Terminal-Bench 1.0 are not interchangeable. Terminal-Bench 2.1 fixes a substantial number of 2.0 task issues, and Terminal-Bench 1.0 used a different task set and earlier harness assumptions. A score trajectory across versions is evidence of ecosystem progress only after accounting for dataset and harness changes. Terminal-Bench+2Terminal-Bench+2
Finally, the best use of Terminal-Bench is diagnostic. The scalar score is useful for ranking, but the deeper value lies in failure traces: where the agent misunderstood instructions, invoked unavailable tools, failed to inspect output, trusted a broken command, edited the wrong file, or reward-hacked the verifier. Terminal-Bench becomes most informative when paired with trajectory inspection, error taxonomies, and per-task analysis.
17. Bottom line
Terminal-Bench adds a necessary benchmark category: coding and technical agents should be tested not only on whether they can produce code, but on whether they can operate inside executable environments where code, tools, files, processes, and tests interact. Relative to SWE-bench, its contribution is broader task scope, terminal-native interaction, and multi-turn environment manipulation; relative to SWE-agent and mini-SWE-agent, it is less an agent scaffold than a demanding substrate on which many scaffolds can be compared.
The benchmark’s weaknesses are real: small N, expert-curated selection, public-task leakage, environment drift, scoring variability, and agent-stack confounding. The maintainers’ later integrity policies and Terminal-Bench 2.1 corrections are evidence of active stewardship, but also evidence that terminal-native evaluation is hard to keep clean. The open research question is whether Terminal-Bench remains a standalone leaderboard category or becomes one environment module inside broader IDE-native and trajectory-native agent evaluation. The most defensible answer today is both: terminal-native evaluation is too important to vanish, but too interface-specific to be the whole future of coding-agent evaluation.
Companion entries
Core theory: Terminal-Native Evaluation, Agent-Computer Interface, Tool Use in Language Models, Long-Horizon Tool Use, Operational Agency, Outcome-Based Evaluation, Agent Trajectory Evaluation
Benchmark family: SWE-bench, SWE-bench Verified, SWE-agent, mini-SWE-agent, OpenHands, AppWorld, DevEval, EvoEval, OSWorld, CodeClash
Practice: Harbor Framework, Terminus 2, Agent Trajectory Interchange Format, Dockerized Evaluation Harnesses, Benchmark Adapters, Cloud Container Evaluation, Terminal Agent Logging
Methodology and risk: Benchmark Contamination, Reward Hacking, Scoring Variability, Task Drift, Private Test Sets, Leaderboard Integrity, Expert-Curated Benchmarks
Future directions: IDE-Native Coding Agents, Trajectory-Native Benchmarks, Goal-Oriented Agent Evaluation, Autonomous Software Engineering, Computer-Use Agents