LLM 작동 원리 시각적 가이드

Skip to main content A Visual Deep Dive How LLMs Actually Work Live LLM Response Human: What is behind this text box? A complete walkthrough of how large language models like ChatGPT are built — from raw internet text to a conversational assistant. Based on Andrej Karpathy's technical deep dive. Training Tokens 15T Parameters 405B Text Data 44 TB Token Vocabulary 100K Scroll to explore Chapter 1 · Pre-Training · Stage 1 Downloading the Internet The first step is collecting an enormous amount of text. Organizations like Common Crawl have been crawling the web since 2007 — indexing 2.7 billion pages by 2024. This raw data is then filtered into a high-quality dataset like FineWeb . The goal: large quantity of high quality , diverse documents. After aggressive filtering, you end up with about 44 terabytes — roughly what fits on a single hard drive — representing ~15 trillion tokens. Key Insight The quality and diversity of this training data has more impact on the final model than almost anything else. Garbage in, garbage out — but at a trillion-token scale. Click any stage to read more detail 🌐 Common Crawl 2.7B web pages · Raw HTML · Since 2007 A non-profit organization that crawls the web and freely provides its data. Their bots follow links from seed pages, recursively indexing the internet. The raw archive is petabytes of gzip'd WARC files containing raw HTML. ↓ 🚫 URL Filtering Blocklists · Malware · Spam · Adult content Block-lists of known malware sites, spam networks, adult content, marketing pages, and low-quality domains are applied. Entire domains can be removed. This is the cheapest filter so it runs first. ↓ 📄 Text Extraction HTML → clean text · Remove navigation & CSS Raw HTML contains <div> tags, CSS, JavaScript, navigation menus, and ads. Parsers extract just the meaningful text content. This is harder than it sounds — heuristics decide what's "content" vs "chrome". ↓ 🌍 Language Filtering Keep pages ≥65% English · Language classifier A language classifier estimates the language of each page. Pages with less than 65% target-language content are dropped. This is a design decision — filter aggressively for one language or train multilingual. ↓ ♻️ Deduplication Exact & fuzzy matching · Reduce repetition Identical or near-identical pages appear millions of times on the internet (copied articles, boilerplate). Training on the same text repeatedly causes memorization. Dedup uses MinHash and exact-match techniques to remove duplicates. ↓ 🔒 PII Removal Names · Addresses · SSNs · Emails Personally Identifiable Information is detected and either redacted or the page is dropped. Regex patterns and ML classifiers find phone numbers, emails, Social Security numbers, physical addresses, and named individuals. ↓ ✅ FineWeb Dataset 44 TB · 15 Trillion tokens · High quality The final filtered dataset. Articles about tornadoes in 2012, medical facts, history, code, recipes, science papers — the full breadth of human knowledge expressed in text. This becomes the training corpus. ▶ Animate Pipeline Chapter 1 · Pre-Training · Stage 2 Tokenization Neural networks can't process raw text — they need numbers. The solution is tokenization : breaking text into "tokens" (sub-word chunks) and assigning each an ID. GPT-4 uses a vocabulary of 100,277 tokens , built via the Byte Pair Encoding (BPE) algorithm. BPE starts with individual bytes (256 symbols), then iteratively merges the most frequent adjacent pairs — compressing the sequence length while expanding the vocabulary. Why not just use words? Words have infinite variants. "run", "running", "runner" would be 3 separate entries. Subword tokens share roots: "run" + "ning", "run" + "ner". This also handles new words, typos, and multiple languages efficiently. BPE in Action BPE Tokenization Steps Interactive diagram showing how Byte Pair Encoding progressively merges characters into subword tokens Next Step → Step 1 of 5 Live Tokenizer Try the examples below or type your own text. Hover (or focus) any token to see its ID. Hello world LLM description Tokenization Numbers Compound words Enter text to tokenize Large language models predict the next token in a sequence. Tokens: 0 Characters: 0 Ratio: 0 chars/token Explore tokenization across GPT-4, Claude, Llama and more → tiktokenizer.vercel.app Chapter 1 · Pre-Training · Stage 3 Training the Neural Network The Transformer neural network is initialized with random parameters — billions of "knobs" . Training adjusts these knobs so the network gets better at predicting the next token in any sequence. Every training step: sample a window of tokens → feed to network → compare prediction to actual next token → nudge all parameters slightly in the right direction. Repeat billions of times . The loss — a single number measuring prediction error — falls steadily as the model learns the statistical patterns of human language. Scale GPT-2 (2019): 1.6B params, 100B tokens, ~$40K to train. Today: same quality for ~$100. Llama 3: 405B params, 15T tokens. Modern frontier models: hundreds of billions of parameters, trillions of tokens. Transformer Architecture Transformer Architecture Select a training stage to see model output quality Step 1 Loss: 11.2 Step 500 Loss: 4.8 Step 5K Loss: 3.1 Step 32K Loss: 2.4 Training Loss ↓ 4.8 Cross-entropy loss 500 Training step Model Output at This Stage the model has learn ing but confus tion still the wqp mxr model bns to predict ... What the model is learning At step 1: pure noise. By step 500: local coherence appears. By step 32K: fluent English. The model is learning grammar, facts, reasoning patterns — all implicitly from token prediction. Chapter 1 · Pre-Training · Stage 4 Inference & Token Sampling Once trained, the network generates text autoregressively: feed a sequence of tokens → get a probability distribution over all 100K possible next tokens → sample one → append → repeat. This process is stochastic — the same prompt generates different outputs every time because we're flipping a biased coin. Higher-probability tokens are more likely but not guaranteed to be chosen. Temperature controls randomness. Low temperature (0.1) → model always picks the top token. High temperature (2.0) → uniform chaos. 0.7–1.0 is the sweet spot for coherent-but-creative text. Key Mental Model The model doesn't "think" about what to say. It computes a probability distribution over all possible next tokens and samples from it. Every word is a coin flip — just a very informed one. Token Sampling Demo Watch the model choose the next word. Each bar shows the probability of a candidate token. The sky appears blue Temperature (randomness) 0.8 Next token candidates Sample Next Token Reset Chapter 2 · The Base Model The Internet Simulator After pre-training, you have a base model — a sophisticated autocomplete engine. It's not an assistant. It doesn't answer questions. It continues token sequences based on what it saw on the internet. Give it a Wikipedia sentence and it'll complete it from memory. Ask it "What is 2+2?" and it might give you a math textbook page, a quiz answer key, or go off on a tangent — whatever was statistically common in its training data. The base model's knowledge lives in its 405 billion parameters — a lossy compression of the internet, like a zip file that approximates rather than perfectly stores information. Base Model Behavior Few-Shot Prompting Hello: Bonjour | Cat: Chat | Dog: Chien | Teacher: → Professeur ✓ correct Memorization Zebras (/ˈzɛbrə, ˈziːbrə/) are African equines with distinctive... ...black-and-white striped coats. There are three living species: the Grévy's zebra, plains zebra, and mountain zebra... ↑ Verbatim Wikipedia recall from weights Hallucination The Republican Party nominated Trump and [running mate] in the 2024 election against... → ...Mike Pence, facing Hillary Clinton and Tim Kaine... → ...Ron DeSantis, against Joe Biden and Kamala Harris... ↑ Knowledge cutoff → plausible confabulation In-Context Learning Ba