신들의 황혼: 11개 LLM의 코드 리팩토링 비교

Twilight of the Gods. Fable and 10 more LLMs on a Code Reorganization Task. Comparison. ¶ Other languages Эта статья также доступна на русском: Гибель богов . Materials & raw data All 11 model proposals, the cross-reviews, the theses runs, and the ranking script are published here: Materials & reproduce this experiment . This is a detailed write-up of one experiment. I took a god node from a real LangGraph agent and asked 5 American and 6 Chinese models first to propose how to untangle it, then to evaluate each other's proposals. After that, I tried three different ways to figure out which of them to trust on the matter. Contents The original problem What a god node is and why it's dangerous What the plan node actually did Lemonade from lemons Why all this, and how the experiment is set up Stage one. The models generate proposals The proposals table A bit more on each proposal Stage two. The models evaluate the proposals The reviews table A bit more on each review Stage three. The wet swimsuit contest Deciding who's good at what Approach one. Do the scores agree? Picking the best proposal Approach two. Comparing reviews by theses Picking the best analyst Approach three. Center of opinion and medoid Picking the best analyst again Deus ex machina . Picking the best analyst one more time Takeaways Which model to use as a generator, which as an evaluator, and where your heart will find peace. The original problem ¶ You know how it goes: you're building a practice AI agent with the fellas on a course by Data Sanity , and amid the colorful whirl of rapidly accreting features you suddenly notice that one of the project's internal agents has a state graph (LangGraph) that looks like this: flowchart TD planner_start([START]) --> plan[plan] plan -->|search| search[search] plan -->|ask_user| ask_user[ask_user / interrupt] plan -->|reflect| reflect[reflect] plan -->|calculate| calculate[calculate] plan -->|finish| finish[finish] search -->|last_observation| observe[observe] search -->|no hits / backend failure| plan observe --> plan calculate --> plan ask_user --> observe_user[observe_user] observe_user --> plan reflect --> plan finish --> planner_end([END]) At first glance this is just a cute little octopus — nothing to worry about. But once you know how much logic this octopus has to hold in its modest eight-legged head, it becomes clear right away that we're looking at an anti-pattern. In this case, let's call it a god node. The plan node hides about 350 lines of logic, including iterative checks, bootstrap questions about region and currency, schema preparation, acquisition-task routing, the LLM call, the subsequent correction of the decision, and so on. The problem isn't just the size of the function. When important orchestration is hidden inside a single node, the graph stops being a representation of the system. It's harder to explain, harder to debug, harder to test, and more dangerous to change. So the obvious task isn't merely to "chop a big function into pieces" but to lift the hidden control logic up to the graph level, so that the resulting architecture becomes clearer and more amenable to further development. What the plan node actually did ¶ The agent this graph was meant to describe was, broadly, in the business of collecting various parameters for downstream calculations. Some of these parameters it cleverly searched for on the internet; some it asked the user about. And it did all this by a not-fully-deterministic algorithm, because depending on the context of a particular conversation, the right way to obtain the same parameter could vary considerably. Here is the set of real functions that had been packed into the plan node: Responsibility What logic was hidden inside plan Iteration loop Incrementing iterations , entering a new planning step, checking status == "aborted" and max_iters Region bootstrap question The _needs_region_question() check and a forced transition to ask_user for core.region Currency bootstrap question The _needs_currency_question() check and a forced transition to ask_user for core.currency Proactive decomposition Generating dynamic_decompositions for fields that need to be broken into components Assembling acquisition recipes Calling build_dynamic_recipes() and preparing the task structure for the subsequent field collection Schema preparation Calling compose_ready_fields() , merging ready component fields into aggregates, and updating schema Calculator limits Checking the calculator attempt limit, a successful or already-current calculation, and other stopping conditions Recovery after a blocked calculation Handling the blocked-calculator scenario, finding the next data-collection task, and, if needed, a fallback decomposition for the problem field General data-collection routing Choosing the next data-collection task without the LLM, including a fast pass over already-opened and component tasks Auto-finish logic Checking whether all source data has been collected, whether any fields remain to act on, and whether the loop can finish without extra steps LLM planning Gathering prompt context, calling _llm().structured(...) , and obtaining a PlannerDecision Post-LLM decomposition Generating an additional decomposition for the model-chosen field if it needs to be broken into components Redirecting and normalizing decisions Redirecting decisions for derived fields, forcing ask_user -> search for fields better found via search, and other deterministic rewrites after the LLM Per-field retries and limits Detecting repeated searches, limits on search and ask_user , and switching to ask_user , reflect , or finish once the limits are exhausted Calculation-decision correction Fixing a premature finish : if the calculation isn't complete yet, the decision is rewritten to calculate or routed into an extra re-evaluation step Bookkeeping state management Resetting or updating decision , decision_origin , llm_failed , status , and other transient flags that accompanied the branching Logging and event dispatch Logging the final decision, emitting progress events, and assembling the final state update before returning from the node Lemonade from lemons ¶ So, we find ourselves in a situation many people will recognize all too well these days. Monsieur Claude has sculpted us a little masterpiece out of spaghetti. Combing out that spaghetti is nowhere near as much fun as bolting on feature after feature with no review whatsoever. But that's only true until you realize you can reduce the entropy with the same tool that raised it. And that's a far more pleasant prospect. But can you trust the untangling of code to the very model that, given half a chance, so gleefully tangles it up? To answer that, I decided to collect several independent architectural proposals from different models, and compare what they'd advise. Eleven models were invited onto the runway for judging: GPT-5.4 GPT-5.5 DeepSeek-4-pro Gemini-3.1-pro GLM-5.1 Kimi-2.6 MiMo-2.5-pro Opus-4.7 Qwen-3.6-plus Qwen-3.7-max Fable-5 First, each of them made its own proposal for splitting plan . Then the models switched into evaluator mode, read the entire set of finished proposals, and ranked them. To make sure I was collecting independent opinions rather than a relay of retellings of one lucky text, the following conditions were enforced: While the proposals were being generated, the models couldn't see one another's work. While the analyses were being generated, they saw every proposal but none of the other analyses. Each run took place in a fresh session. All of the work was done in OpenCode with the Oh My Openagent plugin, at maximum reasoning effort for every model. Stage one. The models generate proposals ¶ In the first stage, each model proposed its own way to lift the plan node's logic up to the graph level. The prompt used to generate each proposal (only the output filename changed): look at docs/planner-graph-ref/current-graph.md. Looks like "plan" n