안스로픽 텍스트 워터마킹, 글쓰기에 대한 무관심 드러내

Earlier this month, Anthropic announced that future versions of Claude will generate text that includes watermarks showing it was AI-generated. At the time, Anthropic did not explain how this would work, leaving us to speculate on the podcast: Would it somehow encode this into the text? Include invisible characters? Do something with the metadata? We now know, thanks to a blog post over the weekend, that Anthropic will do this by changing how its AI writes altogether. “Nothing is added to the text and there are no hidden characters,” Anthropic wrote in that company blog post . “The difference between watermarked and un-watermarked text will not be distinguishable to readers.” The way it will work, the post explained, is that Anthropic will subtly alter the word choices in AI-generated text in a way that is only known to Anthropic and its algorithms. Anthropic will know the watermarking algorithm, which will change “the source of the randomness used to pick among words” and thus can write a tool to detect whether something has been AI-generated. This research and approach is interesting in a data science kind of way, but Anthropic’s layperson explanation for how this will work shows how little the company thinks about the craft of writing or the subtle differences between words a human author might want to use to convey their thoughts. Anthropic asks us to consider the difference between two sentences: “Take the sentence ‘The weather today was cold and…’. The next word is very unlikely to be ‘sugary.’ But it is quite likely to be ‘overcast’ or ‘grey.’ Under most circumstances, it doesn’t matter much to the reader which of these latter two words the model ultimately chooses—the meaning of the sentence is largely the same either way. In cases like this, the choice is settled by a random number,” Anthropic writes. “Watermarking uses low-stakes choices like these—which occur many times over a piece of generated text—to leave a pattern in Claude’s responses. That pattern is undetectable to the reader, but is detectable to anyone who has a key that encodes it. When watermarking is used, choices are still made at random, but the source of the randomness is different.” Anyone who has written anything would, I hope, understand that the difference between the sentences “The weather today was cold and grey” and “The weather today was cold and overcast” are sometimes “low stakes,” as Anthropic describes, but not always. “Grey,” and “overcast” are different words, and there are any number of reasons why a human author might pick one over the other in a given context. In this example, however, Anthropic’s algorithm sees these words as totally interchangeable and thus its watermarking algorithm has decided that it can “nudge” the word choice one way or the other for the purposes of watermarking. Anthropic continues: “Instead of using an arbitrary random number generator to pick the next word, watermarking uses the key and a few words that come before to settle what word the model should pick. That is, the words that Claude picks are still random, but now, one can check the sequence of words and see if it’s consistent with the choices Claude would make if it was using the key. If it is, one can assign a probability that the text was generated by Claude.” Anthropic claims “Watermarking does not impact the quality of Claude’s output. To a reader, a watermarked response is indistinguishable from an unwatermarked one,” and that “in internal testing, we’ve seen no impact of watermarking on the content, level of creativity, or readability of Claude’s text.” People are quite mad about Anthropic’s watermarking system, and understandably so. Synonyms are sometimes interchangeable, but not always, as is pointed out in this excellent essay by John Gruber of Daring Fireball , and by journalism academic Jeff Jarvis, in which he claims Anthropic “devalues writing .” In making this choice, “Anthropic declares words fungible, language random, choice meaningless,” Jarvis writes. When I sat down to write this post, I was mad because it seems like Anthropic is putting its thumb on the scale, messing with the outputs of its machine and saying that the resulting text is qualitatively just the same as the other AI text it was probably going to output. But as I began writing this, I realized that my problem is not necessarily with text watermarking but with AI-generated text altogether. It does not matter to me, necessarily, whether the output of Claude’s garbage AI text is one way or is a slightly different way. But it does matter to me that AI data scientists at huge tech companies think that word choice doesn’t matter, or that it is possible to statistically use synonyms wherever without fucking with the meaning of a sentence. Throughout the blog post, Anthropic describes the act of writing as being akin to a probabilistic game of chance. In Anthropic’s own words, its writing is sometimes the result of an “arbitrary random number generator,” and “random” whenever its systems encounter a situation where its tool believes, based on pattern recognition, that the choice between several possible next words isn’t all that important. That may be true for LLM garbage, but is not true for the human experience of writing, which is why human writing almost always feels different than AI writing. This watermarking approach, and Anthropic’s blog post about it, highlights something that should already be clear about a company that famously scanned and destroyed huge numbers of printed books and has trained its LLMs on stolen content: Anthropic does not care about the craft or effort of writing, and sees words as fungible and unimportant. Anthropic says it is making this change as part of the European Union’s new AI regulations, which are well-intentioned but problematic. While it can definitely be useful to have additional ways of detecting AI-generated content, the carelessness with which Anthropic has announced this decision highlights the broader problem with using LLMs to write: They are, as Anthropic notes, probabilistic tools that do not “write” in the way that humans do, rather, they mimic their training data which is, by definition, things that have already happened and been ingested. Contrast this with how Anthropic sees code, something where it says an “exact output is required.” In writing, meanwhile, Anthropic suggests different words are often “equally good.” Over and over again, Anthropic and the researchers who work on this type of watermarking claim that text can be “nudged” in this way without being noticeable to humans or without impacting “quality.” But it is worth noting that the people judging the “quality” of the AI-generated outputs are either data scientists or people asking AI tools to do their writing for them, not, say, people who care about reading or writing. The scientific paper that Anthropic cites was done by Google researchers on a Google watermarking tool called “SynthID,” which Anthropic’s watermarking is based on. In the SynthID study, quality was assessed by randomly putting watermarking on some Gemini outputs, then asking Gemini users to either thumbs-up or thumbs-down the response: “A random fraction of queries were routed to a watermarked model and an equivalent number to the unwatermarked counterpart. The Gemini user interface allows users to provide feedback on model responses via a thumbs-up (good response) and a thumbs-down (bad response). We analysed approximately 20 million watermarked and unwatermarked responses and computed the thumbs-up and thumbs-down rates (both as a fraction of the total number of thumbs-up and thumbs-down feedback received). We found that the thumbs-up rate for the two models differed by 0.01%.” I hope it is clear to anyone who has clicked on this article that asking someone who asked a chatbot something to thumbs up or thumbs down a response is not a very good way of assessing the “quality” of “writing.” The other human assessme