grep이 LSP를 이기는 이유: 코딩 에이전트는 정밀한 도구를 무시한다

Docs Blog Star 952 Get started All posts Engineering 9 min read Grep beats LSP? Why coding agents ignore your fancier tools I compared grep with LSP-backed semantic navigation across code-finding and editing tasks. The results show why a tool's LLM-friendliness may matter as much as the capability behind it. PX Pengcheng Xu Engineering · 2026-08-12 Copy link Why would a coding agent ignore a retrieval interface that returns more precise results? I explored this question in a small study comparing lexical search with grep against LSP-backed semantic navigation. I expected semantic navigation to reduce noise and save tokens. Instead, agents often stayed with grep . When I forced them to use the semantic path first, task success sometimes fell. This is a question of LLM-friendliness. A tool is not friendly to a model merely because its results are precise. It must return enough context for the next step and present that context in an interface and output shape the model can use directly. Familiarity may also matter: the model may have learned similar action paths during training. The interface properties can be evaluated directly. Training support is a hypothesis consistent with these results, not something this study proves. The result is not a general argument against LSP. The protocol includes capabilities far beyond code navigation, and this study tested only a small subset. Instead, the results point to a broader engineering problem: a model does not use tools in isolation. It uses them through a harness that defines the available actions, their names, their inputs, and the context returned to the model. In this post, I describe how code retrieval affected both code-finding and editing tasks, why grep had an advantage in some conditions, and what this means for agent platforms. Comparing two code retrieval interfaces I compared two ways for an agent to retrieve code context. grep performs lexical search: it finds matching text. The tested LSP-backed tools perform semantic navigation through references, definitions, and document symbols, allowing them to distinguish a real function call from the same word in a comment. The pilot covered three Claude models, several Python and TypeScript repositories, and multiple task types. I measured token use only when both approaches completed the task successfully. This controls for a common evaluation error: a failed run can appear efficient simply because it stopped early. On simple code-location tasks, all three models chose the semantic tool only 0% to 6% of the time when both tools were available. Forcing a semantic-first path reduced success from 100% to 89% in that arm. Reference-completeness tasks produced a different result. When asked to find every caller, the models chose semantic navigation 45% to 57% of the time. The LSP-backed path reached 1.00 precision, compared with 0.76 for grep , by removing false matches. However, recall stayed near 0.66 in both arms. Semantic navigation did not find more true calls. The remaining limit came from how thoroughly the agent worked, not from retrieval precision. For the stronger models, the precision gain also came with higher token use rather than a saving. The model doesn't blindly prefer grep — it routes by task Share of semantic (LSP) tool calls when both grep and LSP are available and the agent chooses freely. Legend: Opus 4.8 (blue), Sonnet 4.6 (magenta), Haiku 4.5 (green). Task Opus 4.8 Sonnet 4.6 Haiku 4.5 Localization 0% 4% 6% Reference-completeness 45% 50% 57% Multi-file rename 3% — — The codebase was also important. On a clean TypeScript repository, LSP-backed navigation produced no F1 gain and used 16% more tokens. On a noisy TypeScript repository, it improved F1 by 0.246 and used 12% fewer tokens. The useful predictor was lexical noise, not whether the language had strong static types. Codebase noise determines the value of semantic navigation Accuracy gain from semantic retrieval on reference-completeness (ΔF1 = LSP − grep). Bar colour encodes how noisy grep is on that repo; prec = grep’s precision there. Legend: blue means grep is clean here; magenta means grep is noisy here. Repo Language grep precision ΔF1 (LSP − grep) Token cost remeda TypeScript 1.00 +0.000 +16% hono TypeScript 0.51 +0.246 −12% requests Python 0.76 +0.072 +19% These results are conditional rather than categorical. The agents did not simply “always use grep.” Their routing changed with the task, and the value of LSP-backed navigation changed with the repository. Tool interfaces change agent behavior The tested LSP-backed tools initially returned only a location: a file path, line, and column. The agent then had to open the file to inspect the code. grep , by contrast, usually returned the matching line immediately: src/auth.ts:42: return validateToken(token) . I changed the semantic-navigation response to include source text in a similar shape. The semantic backend and the set of references stayed the same; only the information returned to the model changed. Pass@1 on the rename tasks rose from 0.67 to 0.83, while follow-up file reads fell from 15.2 to 3.2 per episode. Returning source context improves semantic navigation Multi-file rename, Opus 4.8, pyright with a pre-warmed index. Same semantic backend in both LSP arms — only the output shape differs. Legend: grep (blue), LSP — locations only (magenta), LSP + inline context (green). Arm pass@1 Site recall Tokens Follow-up reads grep 1.00 1.000 2,451 4.3 LSP — locations only 0.67 0.930 4,131 15.2 LSP + inline context 0.83 0.958 3,336 3.2 This result illustrates a principle that Anthropic also emphasizes in Writing effective tools for agents : tools are interfaces for non-deterministic agents, so the context they return is part of the design. A semantically correct tool can still create a poor agent workflow if each result requires several extra actions to interpret. The output change does not prove that post-training data caused the improvement. It may also have helped simply because each response contained more useful information. However, the result is consistent with a broader hypothesis: models learn concrete action patterns, not “tool use” in the abstract. A familiar loop—prompt, tool call, readable result, next action—can be part of the capability observed in practice. Why lexical search had an advantage Interface familiarity is only part of the explanation. Lexical search also had a real structural advantage for some tasks. A semantic reference is only one kind of text match. A rename may also need to update comments, docstrings, configuration, or strings. find_references will not return those by design, while grep can. semantic references ⊂ textual occurrences For text-wide edits, grep can be the better retrieval tool even for a model with perfect training on LSP-backed navigation. This gives us two explanations for the observed behavior: Structure: some tasks need textual completeness, which the tested semantic-navigation methods do not provide. Distribution: the model may have more practice with familiar tools and result shapes. The first explanation follows directly from what the tools retrieve. The second is a hypothesis consistent with the routing and output-format results, but this study did not manipulate training data and therefore cannot prove it. The harness is part of the system Here, I use harness to mean the runtime around a model: the instructions placed in context, the tools made available, their input schemas, the shape of their results and errors, and the loop that decides what the model sees next. This surrounding system can materially change behavior. Anthropic’s work on effective harnesses for long-running agents shows the same idea at a longer time scale: the model alone is not enough to make reliable progress across sessions. Environment setup, progress artifacts, and verification routines affect what the agent can accomplish. The same principle applies within a single tool loop. When post-