거짓 안심을 주는 LLM 자신감 점수

Hello, Reader If someone sent you this post, you have probably tried to extrude a confidence score out of an LLM. I’ve now had this conversation at multiple companies with multiple people, and I’ve had it enough times that it seems like there’s a broader misunderstanding at work here. So my hope is that I can just send people this post instead of relitigating it in a thread every six months. The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless . I want to be upfront that I’d love to be wrong about this. There are people much, much smarter than me working in this space (and I work with many of them!). If you can rebut any of the arguments below, or point me at research I haven’t read, I would genuinely enjoy that conversation. But absent that, these are my current thoughts as I understand them. What I Keep Seeing The pattern shows up everywhere. Chat-based outputs, structured outputs, agentic task results. Someone wants the model to hand back a JSON object with a response key and a confidence key. And more often than not, that confidence key is a continuous score from 0 to 100. That continuous scale is what really grinds my gears, and I’ll come back to why. But the broader problem is that there’s no scientific validity behind any of it. What you have built is a psychological safety trick. It makes the output feel more trustworthy without making it more trustworthy, and I’d argue the people shipping this experience are mostly lying to themselves about what they’ve shipped. LLMs Cannot Reliably Quantify Their Own Confidence I recognize that Anthropic and others have published research arguing that models maintain some kind of latent internal state while they generate. Anthropic’s interpretability work on tracing the thoughts of a language model found that Claude plans several words ahead when writing a rhyming couplet, which is not the behavior you’d predict from a naive next-token story. And their more recent work on emergent introspective awareness found that models can sometimes notice concepts injected into their activations and report on them accurately. That’s real, and it’s interesting. But read the caveat the researchers themselves put on it: the capability is highly unreliable and highly context-dependent. We do not currently have a strong enough understanding of that internal state to assert that models have any usable ability to assess their own correctness. Noticing that something was injected into your activations under laboratory conditions is a very long way from quantifying how likely your paragraph about a customer’s refund policy is to be right. If we think about what’s happening under the hood, there is some quasi-cognition going on in the sense of symbolic relationships between aspects of language. With reasoning models, there’s some observation of that process happening too. But this is where the whole thing collapses for me, because it immediately regresses into a who-watches-the-watchmen problem. Say we want to tap into the thinking trace and use it to evaluate the self-correctness of a response. Fine. That approach has yielded real performance benefits, and I don’t dispute those. But if we’re talking about actually quantifying confidence, the thinking step can’t rescue us, because the obvious next question is: what’s the confidence of the confidence score produced by my thinking step? And then what’s the confidence of that ? It’s confidence all the way down. Do I think reflective reasoning patterns help models catch some of their own errors? Yes. Do I think models currently have a sufficient understanding of their own state to quantify that? No. It’s also worth noting that the research on intrinsic self-correction is not especially encouraging on this front. The DeepMind paper Large Language Models Cannot Self-Correct Reasoning Yet found that when models attempt to correct their initial responses using only their inherent capabilities, without external feedback, performance often degrades . Confident of What, Exactly? Here’s the question I now ask immediately whenever someone requests a confidence score: what is your heuristic for confidence? Because “confidence” gets ambiguous extremely fast. Not slightly ambiguous. Extremely. Is the model confident in the correctness of the response? In the coherence of the response? That it has attempted to fulfill the goal of the user’s request? Those are three completely different questions with three completely different failure modes, and a single float between 0 and 100 flattens all of them into the same number. Absent some heuristic that bounds the concept, these scores are effectively useless. Confidence Isn’t Uniform Across a Response There’s another problem with treating this as a single number, and it goes back to internal state. In traditional machine learning, we have real mechanisms for this. A classifier can output a score that at least serves as a legitimate proxy for confidence, and in a regression setting you can put a prediction interval around an estimate. I want to be careful with that comparison, because classical models don’t hand you a trustworthy number for free either. Guo et al. showed in On Calibration of Modern Neural Networks that modern networks are systematically overconfident, and that raw softmax outputs need post-hoc correction. The difference is that we have an agreed-upon methodology for doing that correction. Platt scaling, isotonic regression, temperature scaling, all fit against held-out ground truth and checked with reliability diagrams and Brier scores. The number earns trust by being measured against reality, and we have a way to check whether that measurement still holds. That machinery is exactly what’s missing when someone adds a confidence key to a JSON schema. The closest analog an LLM has natively is the predicted likelihood of each next token. As a side note, I think “LLMs just lossily predict the next token” has become a real oversimplification of where these systems are, especially once you account for reasoning, adaptive thinking, post-training, and the classifiers running on top of the response. But set that aside and take the most simplistic version. Even there, the problem is that tokens are not uniform. If I’m generating the first hundred words of a response, every single token has its own likelihood value. Some of those are structural glue with near-certain probabilities. Some of them are the load-bearing factual claim in the entire paragraph. Collapsing that distribution into one number that the model then verbalizes is not a measurement. It’s a vibe. This isn’t just my intuition. It’s the reason an entire research line exists to route around raw token probabilities. Semantic entropy, introduced by Farquhar et al. in Nature , samples a model repeatedly, clusters the responses by meaning rather than by wording, and measures the entropy across those meaning clusters. The reason to go to that trouble is that token-level likelihood conflates I could have phrased this five different ways with I don’t actually know this . And note what the method requires: multiple samples and an external comparison step. It’s a measurement performed on the model from the outside, not a number the model reports about itself. I’ve gone looking, and from every piece of research I’ve been able to find, I can’t locate a scenario where these self-reported scores demonstrate the kind of validity people assume they have when they put them in a JSON schema. If You Really Wanted One, Here’s the Work I want to be clear that I’m not claiming it’s impossible to get some reliable indication of confidence out of an LLM. I’m claiming that nobody I’ve met wants it badly enough to do the work required. That work would look something like this. You’d need a strong prompt calibration process in which you define actual heuristics for the model to self-classify against, preferably acr