DeepSWE: 데이터 오염 없는 장기 코딩 에이전트 벤치마크

DeepSWE is a long-horizon software engineering benchmark that delivers four major advances over today's public benchmarks: Contamination free : Tasks are written from scratch, not adapted from existing commits or PRs, so no model has seen the solution during pretraining. High diversity : Tasks span a broad pool of 91 repositories across 5 languages. Real-world complexity : Prompts are half the length of SWE-bench Pro's, yet solutions require 5.5x more code and ~2x more output tokens. Reliable verification : Verifiers are hand-written to test software behavior rather than implementation details. Existing benchmarks fall short on several of these axes. SWE-bench Pro , the leading agentic coding benchmark, has tasks averaging just 120 lines of code to solve, and our audit found its verifier misgrades agent outputs at rates of 8% false positives and 24% false negatives. Frontier labs are also raising growing concerns about benchmark contamination . By contrast, DeepSWE produces a sharper comparison of frontier coding agents. Models that appear close together on public benchmarks separate into wide, ordered gaps that match the differences developers see in day-to-day agent workflows. Leaderboard Models ( 12 / 16 ) gpt-5.5 [ xhigh ] 70 % ± 4 % gpt-5.4 [ xhigh ] 56 % ± 5 % claude-opus-4.7 [ max ] 54 % ± 5 % claude-sonnet-4.6 [ high ] 32 % ± 4 % gemini-3.5-flash [ medium ] 28 % ± 4 % gpt-5.4-mini [ xhigh ] 24 % ± 4 % kimi-k2.6 24 % ± 4 % mimo-v2.5-pro 19 % ± 4 % glm-5.1 18 % ± 4 % gemini-3.1-pro 10 % ± 3 % deepseek-v4-pro 8 % ± 2 % gemini-3-flash 5 % ± 2 % 0 % 20 % 40 % 60 % 80 % gpt-5.5 [ xhigh ] 70 % ± 4 % gpt-5.4 [ xhigh ] 56 % ± 5 % claude-opus-4.7 [ max ] 54 % ± 5 % claude-sonnet-4.6 [ high ] 32 % ± 4 % gemini-3.5-flash [ medium ] 28 % ± 4 % gpt-5.4-mini [ xhigh ] 24 % ± 4 % kimi-k2.6 24 % ± 4 % mimo-v2.5-pro 19 % ± 4 % glm-5.1 18 % ± 4 % gemini-3.1-pro 10 % ± 3 % deepseek-v4-pro 8 % ± 2 % gemini-3-flash 5 % ± 2 % 0 % 25 % 50 % 75 % 100 % All models are run with mini-swe-agent ; see Why mini-swe-agent for a comparison against other harnesses. Explore View the benchmark on GitHub, browse every rollout behind the numbers above, or run your own agent against the benchmark. GitHub → Browse trajectories → Run DeepSWE → Overview 1. Long-horizon work, realistic and short prompts DeepSWE prompts are aligned with the way developers talk to their agents: behavior-focused, short, and free of large interface-definition blocks, rather than overly verbose and prescriptive. Agents must discover where and how to implement the change, so a substantial share of the capabilities being evaluated involve end-to-end exploration instead of just the execution of an overspecified engineering task. Public benchmarks sourced from GitHub issues and pull requests often carry more detail: reproduction steps, additional context, code snippets, and tests that assume specific symbols or signatures. DeepSWE instead scores observable behavior, which lets prompts stay short and natural even when the underlying tasks are substantially longer. DeepSWE tasks are larger in scope and less specified, reflecting real SWE work Mean prompt length 0 1.9k 3.8k 5.6k 7.5k characters SWE-Bench Verified 1,700 SWE-Bench Pro 4,614 DeepSWE 2,158 Mean prompt length SWE-Bench Verified 1,700 SWE-Bench Pro 4,614 DeepSWE 2,158 0 3.8k 7.5k characters Mean reference solution lines added 0 187.5 375 562.5 750 lines added SWE-Bench Verified 10 SWE-Bench Pro 120 DeepSWE 668 Mean reference solution lines added SWE-Bench Verified 10 SWE-Bench Pro 120 DeepSWE 668 0 375 750 lines added Mean files edited per reference solution 0 2.5 5 7.5 10 files SWE-Bench Verified 1 SWE-Bench Pro 5 DeepSWE 7 Mean files edited per reference solution SWE-Bench Verified 1 SWE-Bench Pro 5 DeepSWE 7 0 5 10 files 2. Broad repository coverage DeepSWE contains 113 tasks spanning 91 active open-source repositories across 5 languages: TypeScript, Go, Python, JavaScript, and Rust. Sampling at this scale makes DeepSWE a much stronger proxy for the real-world utilities of coding agents: whether they can make useful, well-scoped changes across varied codebases with different levels of structure, documentation, and maintenance. Existing public benchmarks are much more concentrated. SWE-Bench Pro Public spans 11 repositories, and SWE-Bench Verified spans 12, with many tasks drawn from prominent, heavily maintained projects. That is a narrower setting than the range of projects developers bring to coding agents in practice. Language distribution typescript 35 go 34 python 34 typescript 35 ( 31 %) go 34 ( 30 %) python 34 ( 30 %) javascript 5 ( 4 %) rust 5 ( 4 %) Libraries to large frameworks, across five languages 91 repositories. Dot size is task count; color is primary language. 1k 10k 100k 100 1k 10k GitHub stars Files in default branch TypeScript · 27 Go · 28 Python · 27 JavaScript · 4 Rust · 5 3. Novel tasks test problem-solving, not recall Every DeepSWE task is original: the reference solution is written from scratch rather than copied or adapted from an existing pull request, commit, or public patch. Some tasks are motivated by unresolved GitHub issues, but the fix itself is new. DeepSWE tasks are also never merged back into the upstream repositories, so they do not enter the public GitHub record and are unlikely to appear in future pre-training corpora scraped from open source. This makes DeepSWE a cleaner test of whether an agent can solve a novel software engineering problem, rather than recall, retrieve, or rediscover a public fix. Benchmarks sourced from existing commits have an inherent contamination risk because its corresponding implementation, tests, and discussion are already online. Our SWE-Bench Pro audit found that this risk is not hypothetical, with solution leakage and false positives affecting roughly 8% of audited rollouts. 4. Verifiers reward correctness across many valid implementations A benchmark is only as good as its verifier. In SWE benchmarks, the verifier should approximate the task’s behavioral specification: it should determine whether the submitted code implements the requested change, while remaining agnostic to the particular implementation strategy. DeepSWE’s verifiers are purpose-written from the task description with this goal in mind, and will accept any solution that implements the requested behavior. This differs from a common benchmark construction pattern in which the verifier is inherited from a merged pull request's test suite. These tests provide useful signal, but they are not necessarily designed as complete graders for arbitrary future submissions. They can therefore miss valid solutions, or pass submissions that satisfy the tests without fully satisfying the task. To quantify this, we drew 30 tasks at random from DeepSWE and SWE-Bench Pro and ran 3 rollouts across 10 frontier agent configurations. An LLM then analyzed each trajectory along with the task definition, reference solution and verifier output and then issued an independent verdict on whether the patch actually implemented the requested behavior. The analyzer's verdict could disagree with the verifier in two directions: False positives: the verifier passed but the AI judge concludes the patch does not actually implement the requested behavior. False negatives: the verifier failed but the AI judge concludes the patch is a reasonable solution to the prompt. DeepSWE verifiers align more closely with real task success False positive rate Verifier accepted a wrong implementation 0% 5% 10% SWE-Bench Pro 8.5 % DeepSWE 0.3 % SWE-Bench Pro 8.5 % DeepSWE 0.3 % 0% 5 % 10 % False negative rate Verifier rejected a correct implementation 0% 15% 30% SWE-Bench Pro 24.0 % DeepSWE 1.1 % SWE-Bench Pro 24.0 % DeepSWE 1.1 % 0% 15 % 30 % n = 735 DeepSWE / 789 SWE-Bench Pro reviewed rollouts. Excludes trials with API errors, timeouts, and other transient harness failures from each denominator. Browse false positives SWE-Bench Pro DeepSWE Brow