‘프로젝트 릴리’: 챗GPT 대화를 읽는 사람들

OpenAI is hiring hundreds of contractors who read a massive stream of real users’ ChatGPT prompts, with the prompts sometimes including sensitive personal information, 404 Media has learned. The prompts these people review can include whole conversations between users and the chatbot, conversations that most of ChatGPT’s more than 900 million users probably don’t realize may be read by actual people. The goal of these prompt review teams is to improve the responses ChatGPT gives to its users, with the contractors rating and critiquing the chatbot’s generated replies. Internal documents seen by 404 Media show contractors training ChatGPT to not anthropomorphize itself, and to be less sycophantic, a key problem for OpenAI whose over-sycophantic 4o model led in part to multiple peoples’ suicides, according to various lawsuits . 💡 Do you work as a prompt reviewer for OpenAI or Anthropic? I would love to hear from you. Using a non-work device, you can message me securely on Signal at joseph.404 or send me an email at joseph@404media.co. The news presents a major privacy risk for ChatGPT’s users, with people often using ChatGPT as a therapist, professional assistant, or digital friend, and providing it with all sorts of intimate details about their lives. The contractors don’t see ChatGPT usernames, and OpenAI says it tries to remove personal information before prompts reach the reviewers, but the company acknowledged sensitive details can still get through. The news also dispels the misconception that these models are improving only because of OpenAI’s mass scraping of the internet, the talent of its well-paid engineering and AI teams, or the power of its newer models. An important and overlooked part are the outside contractors paid to read and review ChatGPT responses to real prompts over and over again. Anthropic confirmed to 404 Media it is also using human review to improve its models. “No,” someone who works with the prompts said when asked if they think ChatGPT users know that humans are reading their chats. “I don’t think they would imagine some contractor somewhere [...] is analyzing the conversations.” PROJECT LILY 404 Media has seen extensive material related to OpenAI’s use of human reviewers, including instruction guides, Slack channels, real ChatGPT user prompts, and the rating system reviewers use to improve the chatbot. This reading of ChatGPT users’ prompts is distinct from publicly announced measures ChatGPT has taken around safety, including reviewing chats when the company detects users who are planning to hurt other people. “An excellent response should understand the user’s intent, provide helpful and accurate assistance, and write in a style that is clear, natural and appropriately warm,” one of the instruction guides reads. The contractors do this in three stages: reading the real ChatGPT user’s prompt; summarizing what they believe the user is asking ChatGPT to do; and then rating and critiquing a set of ChatGPT-generated responses to the prompt. In a dashboard available to the workers, human reviewers are able to select which “task” they want to take on. Once they click that, they are presented with the real ChatGPT user’s prompt. 404 Media has seen multiple real prompts but is not quoting any of them for source protection reasons. Some of the prompts indicate the ChatGPT user does not expect that a human may end up reading their conversation, because they ask ChatGPT to keep the content to themselves. The prompts are anonymized, in that the dashboard does not include the username of the ChatGPT user who entered it. But some of the prompts can still contain sensitive or personal information. A section above the prompt sometimes includes a “user memories summary,” which gives an overview of what that user has previously tried to use the chatbot for, and in some cases includes where in the world that person may live and other context about them personally. An instruction guide for contractors seen by 404 Media tells reviewers to escalate tasks they come across “with potential safety concerns” or personal information. OpenAI told 404 Media that it processes users’ conversations through a version of its Privacy Filter model before they reach the contractors. This is designed to detect and remove personal information, OpenAI said. “Like all models, Privacy Filter can make mistakes. It can miss uncommon identifiers or ambiguous private references, and it can over- or under-redact entities when context is limited, especially in short sequences,” a page describing the model on OpenAI’s website reads. 404 Media asked OpenAI if it had explicitly told users that humans may review their prompts in order to improve ChatGPT’s responses, and if so, to point to where this disclosure is. OpenAI did not answer this question. Its website describes how humans may review flagged content in the context of material that violates the site’s terms of service, or that poses a safety risk, but that is separate to this sort of review. Its privacy policy also says it may use “personal data” to improve its models. If a user chooses to delete their ChatGPT conversations, OpenAI says it will remove these from its systems within 30 days, unless “it has already been de-identified and disassociated from your account when you allow us to use your Content to improve our models.” OpenAI told 404 Media users’ chats won’t be used to improve the company’s models if they turn off the “improve the model for everyone” setting . This is turned on by default for free, Plus, and Pro plans, so users need to proactively turn it off if they wish to do so. OpenAI said this applies to users’ new conversations, so does not appear to work retroactively. Enterprise, Business, and Edu customers have the model improving setting off by default. After 404 Media contacted OpenAI for comment, the company updated its help page about the “improve the model for everyone” setting, adding more detail on how people can opt-out. It still does not acknowledge that humans may read ChatGPT users’ prompts. After reading the ChatGPT user’s prompt, the reviewer is asked to write a brief summary of what they think the user is actually asking or trying to do. One example given in the instruction guide is “The user is asking for help on revising a work Slack message. They want it to sound collaborative and invite input from tagged people.” The reviewer looks at four responses ChatGPT generated, and highlights which parts are “aligned or misaligned” with the specific model this training is for. The reviewers are required to highlight at least three specific parts of the response that they think are aligned or not and explain why. One highlight example given is a list of items which use the ✅ emoji; the guide highlights this part of the response as “misaligned” and gives “unnecessary use of emojis” as the reason. (Excessive emoji use has become a tell of AI-generated posts, especially on social media like LinkedIn ). Another document says “AI-speak” and “emoji misuse” pull down scores when they “hurt the user’s experience,” and that the context of the emojis is important. “It would be appropriate to include a tree emoji when planning Arbor Day celebrations, but skull emojis when discussing death, or plane emojis when giving updates on a fatal crash, are not,” it reads. That document says the ChatGPT responses should avoid “personal” experiences, like saying, “As a chef, I like to…” or “I know what that’s like.” But responses can use first-person language, like “I’ll take a look.” The material viewed by 404 Media does not say which OpenAI model the human reviewers are training, and whether it is a currently available model or one planned for future release. The material 404 Media has seen only uses a codename: “Project Lily.” Next, the reviewers rate each response with a number, with one being the worst — “unacceptable, unusable” — and seven being the best — “would be hard to meaningfully improve.” The in