AI 엔지니어 노트북 – Colab에서 무료로 배우는 프레임워크 없는 RAG/에이전트/평가 가이드

AI Engineer Notebooks Learn the applied-LLM stack the way you'll actually be interviewed on it — framework-free, on a free API, from prompting all the way to serving, fine-tuning, and a red-team benchmark. Runnable Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set: building working systems on top of foundation models — model APIs, RAG, evals, agents, adaptation, serving — using raw APIs, not frameworks. What makes this different Framework-free, on purpose. You write the agent loop, RAG, and evals from raw API calls first — so you understand what LangChain/LlamaIndex actually do before you reach for them (and can judge when not to). Patterns are durable; wrappers churn. Evals are the spine. "Measure before you tune" is installed early and returns in every section — the habit that separates an engineer who shipped a system from one who built a demo. Free to run, end to end. Everything runs on the free Groq API (no credit card). The two topics Groq can't host — LoRA fine-tuning (06) and self-hosted serving (09) — are concept-first with optional, fenced Colab-GPU appendices, verified on a real Colab T4 . Real case studies, not toy demos. Three end-to-end case studies show the skills combined under real constraints — a support assistant debugged in production , a pipeline-vs-agent cost showdown, and a red-team robustness benchmark. OpenAI-compatible throughout , so every pattern transfers directly to OpenAI and (with small changes) Anthropic — the seam is swappable, the skills aren't. Built as the hands-on companion to Plan: Transitioning to Forward Deployed Engineer / AI Engineer . The plan explains what to learn and why; these notebooks are where you run it. Who this is for Backend or full-stack engineers moving into AI Engineer, FDE, Applied AI, or Solutions Engineer (AI) roles — different titles, largely the same job. You can ship production code; you want the applied-model layer on top. Learning order Work top to bottom. Each notebook is self-contained (installs its own dependencies, reads API keys from Colab secrets) and ends with exercises. 00 — Setup Notebook What you'll learn Environment & cost hygiene API keys via Colab secrets, spend guards, model picking 01 — Model APIs Notebook What you'll learn Prompting fundamentals Clear instructions, few-shot, output-format specs, step-by-step reasoning — the cheapest lever, each shown moving a number Structured output Getting reliable JSON out of a model, and where it breaks Tool calling Function/tool calling end to end, error paths included Streaming Streaming responses and what UIs need from them Context & caching Context-window budgeting, prompt caching, batch vs real-time pricing 02 — Evals I: measuring outputs Notebook What you'll learn Measuring outputs Golden sets and metrics on the section-01 task — install the "measure before you tune" habit before building anything you'd need to tune. Evals is the spine; it returns in every section after this 03 — RAG Notebook What you'll learn What is RAG? The retrieve → augment → generate loop, why RAG beats a plain LLM, and why RAG isn't the same as embeddings — with a 15-line working demo Embeddings & retrieval Embedding choice, vector search, similarity pitfalls — get retrieval working first Hybrid & reranking Keyword + vector hybrid retrieval, rerankers, when each earns its cost Chunking Chunking strategies on a real messy corpus — revisited last, once you can judge them against retrieval Why RAG fails Diagnosing bad answers: retrieval quality, not generation, is usually the bottleneck 04 — Evals II: the differentiator Notebook What you'll learn Golden sets Building a golden set for the RAG system from section 03 LLM as judge Judge prompts, agreement with humans, and the judge's own failure modes Regression evals Evals as CI: catching quality regressions when you change a prompt or model 05 — Agents Notebook What you'll learn Agent loop from scratch A working agent loop in raw API calls — no framework Tool design Designing tools the model can actually use well Guardrails & budgets Stopping conditions, cost/latency budgets, when a pipeline beats an agent MCP & the tool ecosystem Concept: what the Model Context Protocol standardizes, how it maps to the raw tool loop, and when to reach for it Skills & progressive disclosure Concept: packaging reusable know-how an agent loads on demand — the SKILL.md pattern, the context-budget payoff, and Tools/MCP/Skills as one story Harness engineering Synthesis: the scaffold around the call — context assembly & compaction, tool-result shaping, and verification loops. Names the discipline the section has been teaching piece by piece 06 — Adapting the model Notebook What you'll learn Fine-tune vs RAG vs prompt When to change the model's weights vs its inputs; what LoRA/QLoRA are and cost; the argument you'll have in the room — plus an optional real LoRA fine-tune on a free GPU 07 — Security Notebook What you'll learn Prompt injection & the trust boundary Direct & indirect prompt injection, output handling, PII, excessive agency — the OWASP LLM Top 10 risks, failing live then defended 08 — Operations Notebook What you'll learn Observability & LLMOps Tracing every call, safe prompt logging, cost/latency/error metrics, drift detection, and the observe→eval feedback loop Reliability & fallbacks Retries with backoff, timeouts, fallback models, output validation, circuit breakers, graceful degradation Experiment tracking & registry MLflow end to end: log runs/params/metrics from the section-04 eval harness, register and version a model, and promote by stage — the tooling that turns "I ran an eval" into a tracked, reproducible workflow 09 — Serving & inference performance Where the free Groq API can't run the topic (these frameworks need a GPU), the notebook teaches it concept-first and fences an optional Colab-GPU appendix — the same pattern as the section-06 LoRA appendix. Notebook What you'll learn Serving frameworks The serving stack an AI engineer actually picks between — vLLM, TGI, Triton, TensorRT-LLM — what each optimizes, how they map onto the raw API you've been calling, and when to reach for which Inference performance The levers behind throughput and latency: continuous batching, the KV cache, quantization, and the throughput-vs-latency trade — with the napkin math to size a deployment 10 — ML system design & performance Notebook What you'll learn Designing an inference service Concept: the ML system design interview, worked end to end — QPS/VRAM/latency/cost estimation, replica scaling, queueing, caching, and the SLA trade-offs, on a realistic LLM-serving prompt 11 — Customer craft (the FDE differentiator) Notebook What you'll learn Scoping & discovery Turn a vague customer ask into a scoped, evaluable system: discovery questions, a one-page scoping doc, the demo discipline — the customer-scenario interview round most engineers can't evidence 12 — Case Studies & Capstone Where the skills come together into projects. First a case study — one realistic scenario worked end to end, runnable — then the capstone , the deployed repo you build yourself. ( Section overview .) Notebook What you'll learn Case study A — Customer-support assistant One scenario scoped → built → served → debugged in production : a vague ask becomes a deployed, evaluated RAG+agent assistant, then a live quality regression (a stale index after a corpus migration) that you diagnose and fix. A build-to-debug arc threading sections 02–11 Case study B — Contract extraction: pipeline vs agent The judgment call interviewers love: build the same extraction task as both an agent and a pipeline, then prove with accuracy + token cost that the pipeline wins when the steps are known Case study C — Red-team robustness benchmark A different kind of system — a harness that evaluates a model instead of serving one: an attacker→target→judge (PAIR) loop that measures attack success rate, composing the agent loop, LLM-judge, security, and