메뉴
HN
Hacker News • 11일 전

GPT-5.6 루나 vs GPT-6 아스트라: 1.2달러 모델로 코드 리뷰가 가능할까?

IMP
6/10
핵심 요약

코드 리뷰 벤치마크에서 저가형 GPT-5.6 Luna는 GPT-6 Astra 대비 3.6% 비용으로 75% 수준의 검증된 버그를 찾아냈지만, 보안 버그 탐지율이 크게 뒤떨어지고 오탐률도 4배 이상 높았습니다. 일상적인 정확성 버그에는 Luna로 충분하지만, 인증·권한 코드 리뷰에는 단독 사용이 권장되지 않는다는 결론입니다.

번역된 본문

GPT-5.6 Luna vs GPT-6 Astra: 1.20달러짜리 모델로도 코드 리뷰에 충분할까? Aditya Jha, 2026년 9월 14일

GPT-5.6 Luna는 입력 토큰 100만 개당 0.20달러, 출력 토큰 100만 개당 1.20달러입니다. GPT-6 Astra는 각각 10달러와 50달러입니다. 동일한 풀 리퀘스트(PR)에서 Luna 리뷰 한 번은 0.0041달러, Astra 리뷰 한 번은 0.113달러로 28배 차이가 났습니다.

지난 글에서는 Astra와 GPT-5.6 Sol을 비교했습니다. 이번에는 모든 PR을 가장 저렴한 모델로 검토하면 무엇을 잃게 되는지 알아보고자 했습니다.

간단한 결론 Luna는 50개 PR에서 검증된 버그 69개를 찾았고, Astra는 92개를 찾았습니다. Luna는 전체 실행에 0.20달러, Astra는 5.66달러가 들었습니다. Luna는 틀린 경우가 더 많아 93개 발견 사항 중 24개가 검증을 통과하지 못했고, Astra는 96개 중 4개만 실패했습니다. 보안 버그 24개 중 Luna는 9개, Astra는 19개를 찾았습니다.

우리의 판단: 이 가격이라면 Luna는 일상적인 정확성 버그에는 충분히 좋으며, 인증이나 권한 코드는 Luna 단독으로 리뷰하게 두지 않을 것입니다.

실행 방법 Astra vs Sol 글과 동일한 설정을 재사용해 수치를 비교 가능하게 했습니다. PR은 AI-Code-Review-Evals 조직의 공개 벤치마크 50개로, Cal.com, Sentry, Discourse, Keycloak, Grafana에서 각각 10개씩이며, 깨끗한 베이스 브랜치에 결함을 삽입한 것입니다. Luna와 Astra는 동일한 diff에 동일한 프롬프트를 받았습니다. 프롬프트는 정확성, 보안, 동시성, 리소스, 오류 처리 버그를 요청하며 스타일, 네이밍, 문서, 테스트 제안은 제외합니다. 각 모델은 구조화된 발견 사항을 반환했습니다.

검증은 이전과 동일합니다. 모든 PR마다 Astra, Sol, Luna 및 공개 Entelligence 리뷰어 댓글의 발견 사항을 익명화된 하나의 목록에 넣습니다. GPT-6 Astra와 GPT-5.6 Sol이 각각 diff를 기준으로 해당 목록을 별도로 판정해 중복을 묶고 실제 버그인지 결정합니다. 두 판정자 모두 실제라고 판정해야 검증된 것으로 인정됩니다. 판정자들은 발견 사항의 91%에 대해 일치했고, 143개의 고유 버그가 양쪽을 통과했습니다. Luna의 발견 사항 추가로 판정자가 보는 풀이 달라져 전부 재판정했습니다. Astra의 검증 수는 지난 글의 91에서 92로, Sol은 107에서 108로 변동됐습니다. Astra가 두 판정자 중 하나라 자신에게 약간 유리할 수 있으며, 한계 섹션에서 다룹니다.

결과 Luna는 Astra의 3.6% 비용으로 75%만큼의 검증된 버그를 찾았습니다. 검증된 버그 하나당 Astra는 20배 비쌉니다. Luna는 리뷰당 출력 토큰을 3.1배 더 썼지만 출력 가격이 42배 저렴해 여전히 훨씬 쌌습니다. 리뷰당 36초인 Astra에 비해 23초로 더 빨랐습니다.

팀이 가장 먼저 체감할 것은 정확도 격차입니다. Luna 댓글 4개 중 1개꼴로 틀렸고, Astra는 96개 중 4번만 틀렸습니다. AI 리뷰 댓글을 훑어보는 개발자들은 4분의 1이 노이즈일 때 더 대충 훑게 될 것입니다.

Luna가 뒤처지는 부분 지난 글 독자들은 전체 점수가 한 저장소에서 잘하고 다른 곳에서 못하는 모델을 가릴 수 있다며 코드베이스별, 버그 유형별 결과 분리를 요청했습니다. 이 데이터에서 분리 결과는 Luna가 놓치는 버그의 출처를 보여줍니다.

Sentry, Discourse, Grafana에서 Luna는 Astra와 검증 버그 2개 이내 차이였습니다. Cal.com은 21 대 30으로 격차가 컸고, Keycloak이 가장 컸습니다. Luna는 6개, Astra는 14개의 검증 버그를 찾았고, Keycloak 발견 사항 중 절반만 통과한 반면 Astra는 93%였습니다. Keycloak은 신원·접근 관리 서버로 벤치마크 PR 대부분이 인증 및 권한 로직을 다룹니다.

버그 유형 분할도 같은 방향을 가리킵니다. 모든 검증 버그에 근본 원인별 라벨을 붙였습니다. 여기 비교되는 두 모델이 아닌 GPT-5.6 Sol이 작성된 정의에 따라 143개 버그를 한 번에 라벨링했으며, 라벨은 벤치마크 데이터와 함께 공개돼 누구나 확인할 수 있습니다. 가장 큰 그룹인 데이터·로직 버그에서 Luna는 39개, Astra는 47개를 찾았습니다. 동시성에서는 10 대 13이었고, 보안에서는 24개 중 Luna 9개, Astra 19개였습니다.

Astra가 잡고 Luna가 놓친 Keycloak 버그 두 가지: 연합(Federated) 복구 코드가 사용됨으로 표시되지 않아 하나의 복구 코드를 여러 번 사용할 수 있었습니다.

원문 보기
원문 보기 (영어)
GPT-5.6 Luna vs GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review? Aditya Jha Sep 14, 2026 Aditya Jha Sep 14, 2026 GPT-5.6 Luna costs $0.20 per million input tokens and $1.20 per million output tokens. GPT-6 Astra costs $10 and $50. On the same pull requests, one Luna review cost $0.0041 and one Astra review cost $0.113, a 28x difference. Our last post compared Astra with GPT-5.6 Sol. This time we wanted to know what you give up if every pull request goes through the cheapest model. The short answer Luna found 69 verified bugs across 50 pull requests. Astra found 92. Luna cost $0.20 for the whole run and Astra cost $5.66. Luna was wrong more often, with 24 of its 93 findings failing verification against Astra's 4 of 96, and it found 9 of the 24 security bugs where Astra found 19. Our read: Luna is good enough for everyday correctness bugs at that price, and we wouldn't let it review authentication or permission code on its own. How we ran it We reused the setup from the Astra vs Sol post so the numbers line up. The pull requests are the 50 public benchmark PRs in the AI-Code-Review-Evals organization, ten each from Cal.com, Sentry, Discourse, Keycloak and Grafana. Each one introduces defects against a clean base branch. Luna and Astra got the same prompt on the same diffs. The prompt asks for correctness, security, concurrency, resource and error-handling bugs, and excludes style, naming, docs and test suggestions. Each model returned structured findings. Verification works the same way as before. For every pull request, the findings from Astra, Sol, Luna and the public Entelligence reviewer comments go into one anonymized list. GPT-6 Astra and GPT-5.6 Sol each judge that list separately against the diff, grouping duplicates and deciding whether each issue is a real bug. An issue only counts as verified when both judges call it real. They agreed on 91% of findings, and 143 distinct bugs passed both. Adding Luna's findings changed the pool the judges saw, so everything was judged again. Astra's verified count moved from 91 in the last post to 92 here, and Sol's from 107 to 108. Astra is also one of the two judges, which could favor it slightly. The limits section covers that. The results Luna found 75% as many verified bugs as Astra for 3.6% of the money. Per verified bug, Astra cost 20x more. Luna wrote 3.1x as many output tokens per review as Astra and still came in far cheaper, because its output price is 42x lower. It was also faster, at 23 seconds per review against 36. Your team would feel the precision gap first. About one Luna comment in four was wrong, while Astra was wrong 4 times in 96. Developers who already skim AI review comments will skim harder when a quarter of them are noise. Where Luna falls behind Readers of the last post asked us to split results by codebase and by bug type, because an overall score can hide a model that does well on one repository and badly on another. On this data, the split shows where Luna's missing bugs come from. In Sentry, Discourse and Grafana, Luna came within two verified bugs of Astra. Cal.com had a wider gap, 21 to 30. Keycloak had the widest: Luna found 6 verified bugs to Astra's 14, and only 50% of its Keycloak findings held up, against 93% for Astra. Keycloak is an identity and access management server, and most of its benchmark PRs change authentication and permission logic. The bug-class split points the same way. We labeled every verified bug by root cause. GPT-5.6 Sol, which isn't one of the two models compared here, labeled all 143 bugs in one pass against written definitions. The labels are committed alongside the benchmark data so anyone can check them. On data and logic bugs, the largest group, Luna found 39 to Astra's 47. On concurrency it found 10 to 13. On security, Luna found 9 of 24 and Astra found 19. Two of the Keycloak bugs Astra caught and Luna didn't: Federated recovery codes were never marked as used, so a recovery code could be used more than once. A global view permission overrode denials set on individual clients. Neither looks wrong on any single line. You only see them by working out what the permission model allows after the change. What Luna catches that Astra misses Luna also found bugs Astra missed. Of the 143 verified bugs, 44 were found by both models, 48 only by Astra, and 25 only by Luna. Of the 25 Luna-only bugs, 16 are data and logic bugs and 4 are concurrency bugs. In Discourse, repeating an unsubscribe request kept lowering a user's notification level. In Sentry, a concurrency bug replaced unhealthy worker threads without stopping the old ones. Running both models on every pull request would have found 117 of the 143 verified bugs (82%) for $5.86 in total. That is Luna's $0.20 on top of Astra's $5.66, for 25 more verified bugs. What readers asked us to check Did the models just remember the fixes? One reader pointed out that these repositories are public, and the fixes for the benchmark bugs may sit in their history. A model trained after those fixes landed could be recalling a patch it has already seen. The suggested test was to split the pull requests by date and see whether the ranking holds on changes made after each model's training cutoff. We can't run that split on this benchmark. We pulled the commit date behind every PR, and they range from 2013 to July 25, 2025. 20 are from 2025, and none are recent enough to fall after either model's cutoff. The post-cutoff group would be empty. The risk is smaller than it sounds, because the defects were added to these PRs for the benchmark on purpose, so the exact bug in each diff is not a commit a model could have trained on. The surrounding code is old and public, though, and a model that knows what the correct version looks like has an advantage. Testing that properly needs pull requests newer than the models, and this benchmark can't provide them. Do the models find the same bugs twice? Another reader asked us to rerun some PRs with identical settings. We picked two PRs per codebase and ran each model two more times. From its first run, Astra had 15 verified bugs on those ten PRs. 10 came back in both repeats and 14 in at least one. Luna also had 15. 7 came back in both repeats and 12 in at least one. The sample is small, so treat these as rough. A model that finds a bug on one run can miss it on the next, which applies to every single-run number in this post, and Luna did it more often than Astra. What about bugs nobody flagged? The third request was to track false negatives, meaning real bugs every model missed. Measuring that needs a complete list of the bugs in each PR, which the benchmark doesn't publish. We can give a lower bound. 26 verified bugs were missed by both Luna and Astra and caught only by Sol or the Entelligence reviewer. Two of them are the Discourse security bugs from our last post: a postMessage origin check that used a substring match, and a remote fetch that followed redirects past a host allowlist. The true number of missed bugs is higher, because bugs no reviewer flagged never enter the pool. Limits of this comparison Apart from the ten repeated PRs, each model reviewed each PR once, and the repeat runs show that results move between runs. Astra is both a contestant and one of the two judges. Requiring Sol to agree reduces the bias without removing it. Every PR predates both models' training cutoffs, so the date split readers asked for isn't possible here. Both models saw the diff and nothing else. They had no repository history, call graph, or production data. Verified counts are a floor on the bugs present, and the benchmark has no complete bug list to measure against. What a diff doesn't tell the model On this benchmark, a cheap model did well on most changes and badly on authentication and permission code. A diff alone doesn't tell the model which kind of change it is reviewing. Knowing that a file sits on an authorization path, that a function is called from a login flow, or