Hacker News • 56일 전

SWE-rebench 평가: 13개 모델과 4개 에이전트 벤치마크

IMP

9/10

핵심 요약

소프트웨어 엔지니어링 작업을 평가하는 SWE-rebench v2 리더보드의 최신 결과가 공개되었습니다. 65개 저장소에서 추출된 111개의 문제를 바탕으로 다양한 AI 코딩 모델과 에이전트의 성능, 비용, 토큰 사용량 등을 비교 분석하여 개발 실무에 활용할 수 있는 구체적인 지표를 제공합니다.

번역된 본문

제목: SWE 작업에서의 13개 모델과 4개 에이전트: Go, Java, Python, Rust, TS 소스: 해커뉴스(hackernews)

본문: SWE-rebench 리더보드 ❗ SWE-rebench v2 리더보드 논문 데이터 정보 Discord ❗ SWE-rebench v2 리더보드 논문 데이터 정보 Discord 기간: 2026년 5월 15일 ~ 2026년 7월 1일 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 시작: 2026년 5월 15일 ~ 종료: 2026년 7월 1일

분석 (Insights): 펼치려면 클릭하세요. 모든 에이전트 보기, 모델, 언어 현재 기간 내에 선택된 65개 저장소에서 추출한 총 111개의 문제입니다. 111개 문제, 65개 저장세 검사 (Inspect) 잠재적 데이터 오염, 외부 시스템, 평가 범위 초과

/ 모델 / 해결률 (%) / Pass@5 (%) / 문제당 비용 ($) / 문제당 토큰 수

1 Anthropic Fable 5 [high] / 모델 / 64.5% ± 1.41% / 78.4% / $4.40 / 2,518,308 (94.9% 캐시됨) 2 Grok Grok 4.5 [high] / 모델 / 63.8% ± 0.60% / 77.5% / $1.47 / 2,429,424 (92.6% 캐시됨) 3 Anthropic Opus 5 [high] / 모델 / 63.4% ± 1.35% / 74.8% / $3.47 / 4,322,143 (95.7% 캐시됨) 4 Z.ai GLM-5.2 [high] / 모델 / 62.9% ± 1.19% / 81.1% / $1.40 / 5,524,892 (92.1% 캐시됨) 5 OpenAI GPT-5.6 Sol [medium] / 모델 / 62.3% ± 1.83% / 79.3% / $0.85 / 605,340 (84.7% 캐시됨) 6 Junie Junie / 에이전트 / 61.8% ± 0.54% / 73.9% / $0.81 / 1,684,501 (85.9% 캐시됨) 7 Anthropic Claude Code / 에이전트 / 60.4% ± 1.03% / 75.7% / $3.39 / 3,341,581 (93.4% 캐시됨) 8 OpenAI Codex / 에이전트 / 58.0% ± 1.29% / 73.0% / $1.59 / 2,070,976 (94.6% 캐시됨) 9 Anthropic Sonnet 5 [high] / 모델 / 56.8% ± 0.94% / 74.8% / $1.43 / 4,645,617 (96.4% 캐시됨) 10 Cursor Cursor / 에이전트 / 51.7% ± 0.84% / 65.8% / $0.41 / 1,827,002 (94.3% 캐시됨) 11 Minimax MiniMax M3 / 모델 / 47.2% ± 1.13% / 69.4% / $0.95 / 13,869,459 (97.0% 캐시됨) 12 XiaomiMiMo MiMo V2.5 Pro / 모델 / 46.5% ± 0.54% / 65.8% / $0.10 / 4,687,987 (95.7% 캐시됨) 13 OpenAI GPT-5.6 Luna [medium] / 모델 / 43.6% ± 1.47% / 59.5% / $0.11 / 395,522 (85.2% 캐시됨) 14 DeepSeek DeepSeek-V4 Pro [high] / 모델 / 40.2% ± 1.29% / 64.0% / $0.15 / 3,955,414 (91.2% 캐시됨) 15 Qwen Qwen3.6-27B / 모델 / 31.2% ± 1.68% / 57.7% / $0.62 / 3,078,970 (76.8% 캐시됨) 16 Qwen Qwen3.6-35B-A3B / 모델 / 24.7% ± 0.79% / 43.2% / $0.27 / 3,726,061 (87.5% 캐시됨) 17 Qwen Qwen3.5-35B-A3B / 모델 / 17.1% ± 1.07% / 36.9% / $0.99 / 5,939,227 (72.3% 캐시됨)

18 Anthropic Claude Opus 4.1 / 모델 / 데이터 없음 (N/A) 19 Anthropic Claude Opus 4.5 / 모델 / 데이터 없음 (N/A) 20 Anthropic Claude Opus 4.6-high / 모델 / 데이터 없음 (N/A) 21 Anthropic Claude Opus 4.7-high / 모델 / 데이터 없음 (N/A) 22 Anthropic Claude Opus 4.8-xhigh / 모델 / 데이터 없음 (N/A) 23 Anthropic Claude Sonnet 3.5 / 모델 / 데이터 없음 (N/A) 24 Anthropic Claude Sonnet 4 / 모델 / 데이터 없음 (N/A) 25 Anthropic Claude Sonnet 4.5 / 모델 / 데이터 없음 (N/A) 26 Anthropic Claude Sonnet 4.6 / 모델 / 데이터 없음 (N/A) 27 DeepSeek DeepSeek-R1-0528 / 모델 / 데이터 없음 (N/A) 28 DeepSeek DeepSeek-V3 / 모델 / 데이터 없음 (N/A) 29 DeepSeek DeepSeek-V3-0324 / 모델 / 데이터 없음 (N/A) 30 DeepSeek DeepSeek-V3-0324 / 모델 / 데이터 없음 (N/A) 31 DeepSeek DeepSeek-V3.1 / 모델 / 데이터 없음 (N/A) 32 DeepSeek DeepSeek-V3.2 / 모델 / 데이터 없음 (N/A) 33 DeepSeek DeepSeek-V4 Flash [high] / 모델 / 데이터 없음 (N/A) 34 Mistral Devstral-2-123B-Instruct-2512 / 모델 / 데이터 없음 (N/A) 35 Mistral Devstral-Small-2-24B-Instruct-2512 / 모델 / 데이터 없음 (N/A) 36 Mistral Devstral-Small-2505 / 모델 / 데이터 없음 (N/A) 37 Gemini Gemini 3 Flash Preview / 모델 / 데이터 없음 (N/A) 38 Gemini Gemini 3 Pro Preview / 모델 / 데이터 없음 (N/A) 39 Gemini Gemini 3.1 Pro Preview / 모델 / 데이터 없음 (N/A) 40 Gemini Gemini 3.5 Flash / 모델 / 데이터 없음 (N/A) 41 Gemini gemini-2.0-flash / 모델 / 데이터 없음 (N/A) 42 Gemini gemini-2.0-flash / 모델 / 데이터 없음 (N/A) 43 Gemini gemini-2.5-flash / 모델 / 데이터 없음 (N/A) 44 Gemini gemini-2.5-flash-preview-05-20 no-thinking / 모델 / 데이터 없음 (N/A) 45 Gemini gemini-2.5-flash-preview-05-20 no-thinking / 모델 / 데이터 없음 (N/A) 46 Gemini gemini-2.5-pro / 모델 / 데이터 없음 (N/A) 47 Gemini Gemma 4 31B / 모델 / 데이터 없음 (N/A) 48 Gemini gemma-3-27b-it / 모델 / 데이터 없음 (N/A) 49 Z.ai GLM-4.5 / 모델 / 데이터 없음 (N/A) 50 Z.ai GLM-4.5 Air / 모델 / 데이터 없음 (N/A) 51 Z.ai GLM-4.6 / 모델 / 데이터 없음 (N/A) 52 Z.ai GLM-4.7 / 모델 / 데이터 없음 (N/A) 53 Z.ai GLM-4.7 Flash / 모델 / 데이터 없음 (N/A) 54 Z.ai GLM-5 / 모델 / 데이터 없음 (N/A) 55 Z.ai GLM-5.1 / 모델 / 데이터 없음 (N/A) 56 Z.ai GLM-5.1 / 모델 / 데이터 없음 (N/A) 57 OpenAI gpt-4.1-2025-04-14 / 모델 / 데이터 없음 (N/A) 58 Open (이하 생략)

원문 보기

원문 보기 (영어)

SWE-rebench Leaderboard ❗ SWE-rebench v2 Leaderboard Paper Data About Discord ❗ SWE-rebench v2 Leaderboard Paper Data About Discord Time Window 15/05/2026 01/07/2026 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 From 15/05/2026 To 01/07/2026 Insights Click to expand Insights View All Agent Model Language All 111 problems from 65 repositories selected within the current time window. Inspect 111 problems, 65 repositories Inspect Potential contamination External system Beyond eval range # Model Resolved Rate (%) Pass@5 (%) Cost per Problem ($) Tokens per Problem 1 Anthropic Fable 5 [high] Model 64.5% ± 1.41% 78.4% $4.40 2,518,308 94.9% cached 2 Grok Grok 4.5 [high] Model 63.8% ± 0.60% 77.5% $1.47 2,429,424 92.6% cached 3 Anthropic Opus 5 [high] Model 63.4% ± 1.35% 74.8% $3.47 4,322,143 95.7% cached 4 Z.ai GLM-5.2 [high] Model 62.9% ± 1.19% 81.1% $1.40 5,524,892 92.1% cached 5 OpenAI GPT-5.6 Sol [medium] Model 62.3% ± 1.83% 79.3% $0.85 605,340 84.7% cached 6 Junie Junie Agent 61.8% ± 0.54% 73.9% $0.81 1,684,501 85.9% cached 7 Anthropic Claude Code Agent 60.4% ± 1.03% 75.7% $3.39 3,341,581 93.4% cached 8 OpenAI Codex Agent 58.0% ± 1.29% 73.0% $1.59 2,070,976 94.6% cached 9 Anthropic Sonnet 5 [high] Model 56.8% ± 0.94% 74.8% $1.43 4,645,617 96.4% cached 10 Cursor Cursor Agent 51.7% ± 0.84% 65.8% $0.41 1,827,002 94.3% cached 11 Minimax MiniMax M3 Model 47.2% ± 1.13% 69.4% $0.95 13,869,459 97.0% cached 12 XiaomiMiMo MiMo V2.5 Pro Model 46.5% ± 0.54% 65.8% $0.10 4,687,987 95.7% cached 13 OpenAI GPT-5.6 Luna [medium] Model 43.6% ± 1.47% 59.5% $0.11 395,522 85.2% cached 14 DeepSeek DeepSeek-V4 Pro [high] Model 40.2% ± 1.29% 64.0% $0.15 3,955,414 91.2% cached 15 Qwen Qwen3.6-27B Model 31.2% ± 1.68% 57.7% $0.62 3,078,970 76.8% cached 16 Qwen Qwen3.6-35B-A3B Model 24.7% ± 0.79% 43.2% $0.27 3,726,061 87.5% cached 17 Qwen Qwen3.5-35B-A3B Model 17.1% ± 1.07% 36.9% $0.99 5,939,227 72.3% cached 18 Anthropic Claude Opus 4.1 Model N/A N/A N/A N/A 19 Anthropic Claude Opus 4.5 Model N/A N/A N/A N/A 20 Anthropic Claude Opus 4.6-high Model N/A N/A N/A N/A 21 Anthropic Claude Opus 4.7-high Model N/A N/A N/A N/A 22 Anthropic Claude Opus 4.8-xhigh Model N/A N/A N/A N/A 23 Anthropic Claude Sonnet 3.5 Model N/A N/A N/A N/A 24 Anthropic Claude Sonnet 4 Model N/A N/A N/A N/A 25 Anthropic Claude Sonnet 4.5 Model N/A N/A N/A N/A 26 Anthropic Claude Sonnet 4.6 Model N/A N/A N/A N/A 27 DeepSeek DeepSeek-R1-0528 Model N/A N/A N/A N/A 28 DeepSeek DeepSeek-V3 Model N/A N/A N/A N/A 29 DeepSeek DeepSeek-V3-0324 Model N/A N/A N/A N/A 30 DeepSeek DeepSeek-V3-0324 Model N/A N/A N/A N/A 31 DeepSeek DeepSeek-V3.1 Model N/A N/A N/A N/A 32 DeepSeek DeepSeek-V3.2 Model N/A N/A N/A N/A 33 DeepSeek DeepSeek-V4 Flash [high] Model N/A N/A N/A N/A 34 Mistral Devstral-2-123B-Instruct-2512 Model N/A N/A N/A N/A 35 Mistral Devstral-Small-2-24B-Instruct-2512 Model N/A N/A N/A N/A 36 Mistral Devstral-Small-2505 Model N/A N/A N/A N/A 37 Gemini Gemini 3 Flash Preview Model N/A N/A N/A N/A 38 Gemini Gemini 3 Pro Preview Model N/A N/A N/A N/A 39 Gemini Gemini 3.1 Pro Preview Model N/A N/A N/A N/A 40 Gemini Gemini 3.5 Flash Model N/A N/A N/A N/A 41 Gemini gemini-2.0-flash Model N/A N/A N/A N/A 42 Gemini gemini-2.0-flash Model N/A N/A N/A N/A 43 Gemini gemini-2.5-flash Model N/A N/A N/A N/A 44 Gemini gemini-2.5-flash-preview-05-20 no-thinking Model N/A N/A N/A N/A 45 Gemini gemini-2.5-flash-preview-05-20 no-thinking Model N/A N/A N/A N/A 46 Gemini gemini-2.5-pro Model N/A N/A N/A N/A 47 Gemini Gemma 4 31B Model N/A N/A N/A N/A 48 Gemini gemma-3-27b-it Model N/A N/A N/A N/A 49 Z.ai GLM-4.5 Model N/A N/A N/A N/A 50 Z.ai GLM-4.5 Air Model N/A N/A N/A N/A 51 Z.ai GLM-4.6 Model N/A N/A N/A N/A 52 Z.ai GLM-4.7 Model N/A N/A N/A N/A 53 Z.ai GLM-4.7 Flash Model N/A N/A N/A N/A 54 Z.ai GLM-5 Model N/A N/A N/A N/A 55 Z.ai GLM-5.1 Model N/A N/A N/A N/A 56 Z.ai GLM-5.1 Model N/A N/A N/A N/A 57 OpenAI gpt-4.1-2025-04-14 Model N/A N/A N/A N/A 58 OpenAI gpt-4.1-2025-04-14 Model N/A N/A N/A N/A 59 OpenAI gpt-4.1-mini-2025-04-14 Model N/A N/A N/A N/A 60 OpenAI gpt-4.1-mini-2025-04-14 Model N/A N/A N/A N/A 61 OpenAI gpt-4.1-nano-2025-04-14 Model N/A N/A N/A N/A 62 OpenAI gpt-5-2025-08-07-high Model N/A N/A N/A N/A 63 OpenAI gpt-5-2025-08-07-medium Model N/A N/A N/A N/A 64 OpenAI gpt-5-2025-08-07-minimal Model N/A N/A N/A N/A 65 OpenAI gpt-5-codex Model N/A N/A N/A N/A 66 OpenAI gpt-5-mini-2025-08-07-high Model N/A N/A N/A N/A 67 OpenAI gpt-5-mini-2025-08-07-medium Model N/A N/A N/A N/A 68 OpenAI gpt-5.1-codex Model N/A N/A N/A N/A 69 OpenAI gpt-5.1-codex-max Model N/A N/A N/A N/A 70 OpenAI gpt-5.2-2025-12-11-medium Model N/A N/A N/A N/A 71 OpenAI gpt-5.2-2025-12-11-xhigh Model N/A N/A N/A N/A 72 OpenAI gpt-5.2-codex Model N/A N/A N/A N/A 73 OpenAI gpt-5.3-codex Model N/A N/A N/A N/A 74 OpenAI gpt-5.3-codex-xhigh Model N/A N/A N/A N/A 75 OpenAI gpt-5.4-2026-03-05-medium Model N/A N/A N/A N/A 76 OpenAI gpt-5.5-2026-04-23-medium Model N/A N/A N/A N/A 77 OpenAI gpt-5.5-2026-04-23-xhigh Model N/A N/A N/A N/A 78 OpenAI gpt-oss-120b Model N/A N/A N/A N/A 79 OpenAI gpt-oss-120b-high Model N/A N/A N/A N/A 80 OpenAI gpt-oss-20b Model N/A N/A N/A N/A 81 Grok Grok 4 Model N/A N/A N/A N/A 82 Grok Grok Code Fast 1 Model N/A N/A N/A N/A 83 OpenRouter horizon-alpha Model N/A N/A N/A N/A 84 OpenRouter horizon-beta Model N/A N/A N/A N/A 85 Kimi Kimi K2 Model N/A N/A N/A N/A 86 Kimi Kimi K2 Instruct 0905 Model N/A N/A N/A N/A 87 Kimi Kimi K2 Thinking Model N/A N/A N/A N/A 88 Kimi Kimi K2.5 Model N/A N/A N/A N/A 89 Kimi Kimi K2.6 Model N/A N/A N/A N/A 90 Meta Llama-3.3-70B-Instruct Model N/A N/A N/A N/A 91 Meta Llama-4-Maverick-17B-128E-Instruct Model N/A N/A N/A N/A 92 Meta Llama-4-Scout-17B-16E-Instruct Model N/A N/A N/A N/A 93 Minimax MiniMax M2 Model N/A N/A N/A N/A 94 Minimax MiniMax M2.1 Model N/A N/A N/A N/A 95 Minimax MiniMax M2.5 Model N/A N/A N/A N/A 96 Minimax MiniMax M2.7 Model N/A N/A N/A N/A 97 OpenAI o3-2025-04-16 Model N/A N/A N/A N/A 98 OpenAI o4-mini-2025-04-16 Model N/A N/A N/A N/A 99 Qwen Qwen2.5-72B-Instruct Model N/A N/A N/A N/A 100 Qwen Qwen2.5-Coder-32B-Instruct Model N/A N/A N/A N/A 101 Qwen Qwen3-235B-A22B Model N/A N/A N/A N/A 102 Qwen Qwen3-235B-A22B no-thinking Model N/A N/A N/A N/A 103 Qwen Qwen3-235B-A22B thinking Model N/A N/A N/A N/A 104 Qwen Qwen3-235B-A22B-Instruct-2507 Model N/A N/A N/A N/A 105 Qwen Qwen3-235B-A22B-Thinking-2507 Model N/A N/A N/A N/A 106 Qwen Qwen3-30B-A3B-Instruct-2507 Model N/A N/A N/A N/A 107 Qwen Qwen3-30B-A3B-Thinking-2507 Model N/A N/A N/A N/A 108 Qwen Qwen3-32B Model N/A N/A N/A N/A 109 Qwen Qwen3-32B no-thinking Model N/A N/A N/A N/A 110 Qwen Qwen3-32B thinking Model N/A N/A N/A N/A 111 Qwen Qwen3-Coder-30B-A3B-Instruct Model N/A N/A N/A N/A 112 Qwen Qwen3-Coder-480B-A35B-Instruct Model N/A N/A N/A N/A 113 Qwen Qwen3-Coder-Next Model N/A N/A N/A N/A 114 Qwen Qwen3-Next-80B-A3B-Instruct Model N/A N/A N/A N/A 115 Qwen Qwen3.5-27B Model N/A N/A N/A N/A 116 Qwen Qwen3.5-397B-A17B Model N/A N/A N/A N/A 117 Stepfun Step-3.5-Flash Model N/A N/A N/A N/A News [2026-07-01]: Added new models to the leaderboad: GLM 5.2, DeepSeek-V4 Pro, DeepSeek-V4 Flash, MiMo V2.5 Pro, Qwen3.6-35B-A3B, Qwen3.6-27B and Gemma 4 31B. [2026-06-09]: Added new models to the leaderboad: Gemini 3.5 Flash and MiniMax M3. [2026-05-28]: Added new models to the leaderboad: Claude Opus 4.8. [2026-05-27]: Added new models to the leaderboad: gpt-5.5-2026-04-23-xhigh, gpt-5.5-2026-04-23-medium, gpt-5.4-2026-03-05-medium, Claude Opus 4.7, and Kimi K2.6. [2026-04-19]: Re-run the Junie with Claude Opus 4.6 as the primary model. [2026-04-15]: Added new models to the leaderboad: GLM-5.1, Qwen3.5-27B, Cursor, Gemma 4 31B and MiniMax M2.7. [2026-03-20]: Added new models to the leaderboard: gpt-5.4-2026-03-05-medium, Gemini 3.1 Pro Preview, Claude Sonnet 4.6, Qwen3.5-397B-A17B, gpt-5.3-codex-xhigh, gpt-5.3-codex and Qwen3.5-35B-A3B Deprecated following models: gpt-5.2-2025-12-11-xhigh,

벤치마크 코딩에이전트 AI모델 소프트웨어개발