초경량 1비트 라마 모델 'Bonsai' CUDA 활용 실전 튜토리얼

Technology AI Shorts Artificial Intelligence Editors Pick Language Model Staff Tutorials In this tutorial, we implement how to run the Bonsai 1-bit large language model efficiently using GPU acceleration and PrismML’s optimized GGUF deployment stack. We set up the environment, install the required dependencies, and download the prebuilt llama.cpp binaries, and load the Bonsai-1.7B model for fast inference on CUDA. As we progress, we examine how 1-bit quantization works under the hood, why the Q1_0_g128 format is so memory-efficient, and how this makes Bonsai practical for lightweight yet capable language model deployment. We also test core inference, benchmarking, multi-turn chat, structured JSON generation, code generation, OpenAI-compatible server mode, and a small retrieval-augmented generation workflow, giving us a complete, hands-on view of how Bonsai operates in real-world use. Copy Code Copied Use a different Browser import os, sys, subprocess, time, json, urllib.request, tarfile, textwrap try: import google.colab IN_COLAB = True except ImportError: IN_COLAB = False def section(title): bar = "═" * 60 print(f"\n{bar}\n {title}\n{bar}") section("1 · Environment & GPU Check") def run(cmd, capture=False, check=True, **kw): return subprocess.run( cmd, shell=True, capture_output=capture, text=True, check=check, **kw ) gpu_info = run("nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader", capture=True, check=False) if gpu_info.returncode == 0: print("✅ GPU detected:", gpu_info.stdout.strip()) else: print("⚠️ No GPU found — inference will run on CPU (much slower).") cuda_check = run("nvcc --version", capture=True, check=False) if cuda_check.returncode == 0: for line in cuda_check.stdout.splitlines(): if "release" in line: print(" CUDA:", line.strip()) break print(f" Python {sys.version.split()[0]} | Platform: Linux (Colab)") section("2 · Installing Python Dependencies") run("pip install -q huggingface_hub requests tqdm openai") print("✅ huggingface_hub, requests, tqdm, openai installed") from huggingface_hub import hf_hub_download We begin by importing the core Python modules that we need for system operations, downloads, timing, and JSON handling. We check whether we are running inside Google Colab, define a reusable section printer, and create a helper function to run shell commands cleanly from Python. We then verify the GPU and CUDA environment, print the Python runtime details, install the required Python dependencies, and prepare the Hugging Face download utility for the next stages. Copy Code Copied Use a different Browser section("3 · Downloading PrismML llama.cpp Prebuilt Binaries") RELEASE_TAG = "prism-b8194-1179bfc" BASE_URL = f"https://github.com/PrismML-Eng/llama.cpp/releases/download/{RELEASE_TAG}" BIN_DIR = "/content/bonsai_bin" os.makedirs(BIN_DIR, exist_ok=True) def detect_cuda_build(): r = run("nvcc --version", capture=True, check=False) for line in r.stdout.splitlines(): if "release" in line: try: ver = float(line.split("release")[-1].strip().split(",")[0].strip()) if ver >= 13.0: return "13.1" if ver >= 12.6: return "12.8" return "12.4" except ValueError: pass return "12.4" cuda_build = detect_cuda_build() print(f" Detected CUDA build slot: {cuda_build}") TAR_NAME = f"llama-{RELEASE_TAG}-bin-linux-cuda-{cuda_build}-x64.tar.gz" TAR_URL = f"{BASE_URL}/{TAR_NAME}" tar_path = f"/tmp/{TAR_NAME}" if not os.path.exists(f"{BIN_DIR}/llama-cli"): print(f" Downloading: {TAR_URL}") urllib.request.urlretrieve(TAR_URL, tar_path) print(" Extracting …") with tarfile.open(tar_path, "r:gz") as t: t.extractall(BIN_DIR) for fname in os.listdir(BIN_DIR): fp = os.path.join(BIN_DIR, fname) if os.path.isfile(fp): os.chmod(fp, 0o755) print(f"✅ Binaries extracted to {BIN_DIR}") bins = sorted(f for f in os.listdir(BIN_DIR) if os.path.isfile(os.path.join(BIN_DIR, f))) print(" Available:", ", ".join(bins)) else: print(f"✅ Binaries already present at {BIN_DIR}") LLAMA_CLI = f"{BIN_DIR}/llama-cli" LLAMA_SERVER = f"{BIN_DIR}/llama-server" test = run(f"{LLAMA_CLI} --version", capture=True, check=False) if test.returncode == 0: print(f" llama-cli version: {test.stdout.strip()[:80]}") else: print(f"⚠️ llama-cli test failed: {test.stderr.strip()[:200]}") section("4 · Downloading Bonsai-1.7B GGUF Model") MODEL_REPO = "prism-ml/Bonsai-1.7B-gguf" MODEL_DIR = "/content/bonsai_models" GGUF_FILENAME = "Bonsai-1.7B.gguf" os.makedirs(MODEL_DIR, exist_ok=True) MODEL_PATH = os.path.join(MODEL_DIR, GGUF_FILENAME) if not os.path.exists(MODEL_PATH): print(f" Downloading {GGUF_FILENAME} (~248 MB) from HuggingFace …") MODEL_PATH = hf_hub_download( repo_id=MODEL_REPO, filename=GGUF_FILENAME, local_dir=MODEL_DIR, ) print(f"✅ Model saved to: {MODEL_PATH}") else: print(f"✅ Model already cached: {MODEL_PATH}") size_mb = os.path.getsize(MODEL_PATH) / 1e6 print(f" File size on disk: {size_mb:.1f} MB") section("5 · Core Inference Helpers") DEFAULT_GEN_ARGS = dict( temp=0.5, top_p=0.85, top_k=20, repeat_penalty=1.0, n_predict=256, n_gpu_layers=99, ctx_size=4096, ) def build_llama_cmd(prompt, system_prompt="You are a helpful assistant.", **overrides): args = {**DEFAULT_GEN_ARGS, **overrides} formatted = ( f"<|im_start|>system\n{system_prompt}<|im_end|>\n" f"<|im_start|>user\n{prompt}<|im_end|>\n" f"<|im_start|>assistant\n" ) safe_prompt = formatted.replace('"', '\\"') return ( f'{LLAMA_CLI} -m "{MODEL_PATH}"' f' -p "{safe_prompt}"' f' -n {args["n_predict"]}' f' --temp {args["temp"]}' f' --top-p {args["top_p"]}' f' --top-k {args["top_k"]}' f' --repeat-penalty {args["repeat_penalty"]}' f' -ngl {args["n_gpu_layers"]}' f' -c {args["ctx_size"]}' f' --no-display-prompt' f' -e' ) def infer(prompt, system_prompt="You are a helpful assistant.", verbose=True, **overrides): cmd = build_llama_cmd(prompt, system_prompt, **overrides) t0 = time.time() result = run(cmd, capture=True, check=False) elapsed = time.time() - t0 output = result.stdout.strip() if verbose: print(f"\n{'─'*50}") print(f"Prompt : {prompt[:100]}{'…' if len(prompt) > 100 else ''}") print(f"{'─'*50}") print(output) print(f"{'─'*50}") print(f"⏱ {elapsed:.2f}s | ~{len(output.split())} words") return output, elapsed print("✅ Inference helpers ready.") section("6 · Basic Inference — Hello, Bonsai!") infer("What makes 1-bit language models special compared to standard models?") We download and prepare the PrismML prebuilt llama.cpp CUDA binaries that power local inference for the Bonsai model. We detect the available CUDA version, choose the matching binary build, extract the downloaded archive, make the files executable, and verify that the llama-cli binary works correctly. After that, we download the Bonsai-1.7B GGUF model from Hugging Face, set up the model path, define the default generation settings, and build the core helper functions that format prompts and run inference. Copy Code Copied Use a different Browser section("7 · Q1_0_g128 Quantization — What's Happening Under the Hood") print(textwrap.dedent(""" ╔══════════════════════════════════════════════════════════════╗ ║ Bonsai Q1_0_g128 Weight Representation ║ ╠══════════════════════════════════════════════════════════════╣ ║ Each weight = 1 bit: 0 → −scale ║ ║ 1 → +scale ║ ║ Every 128 weights share one FP16 scale factor. ║ ║ ║ ║ Effective bits per weight: ║ ║ 1 bit (sign) + 16/128 bits (shared scale) = 1.125 bpw ║ ║ ║ ║ Memory comparison for Bonsai-1.7B: ║ ║ FP16: 3.44 GB (1.0× baseline) ║ ║ Q1_0_g128: 0.24 GB (14.2× smaller!) ║ ║ MLX 1-bit g128: 0.27 GB (12.8× smaller) ║ ╚══════════════════════════════════════════════════════════════╝ """)) print("📐 Python demo of Q1_0_g128 quantization logic:\n") import random random.seed(42) GROUP_SIZE = 128 weights_fp16 = [random.gauss(0, 0.1) for _ in range(GROUP_SIZE)] scale = max(abs(w) for w in weights_fp16) quantized = [1 if w >= 0 else 0 for w in weights_fp16] dequantized = [scale if b == 1 else -scale for b in quantized] mse = sum((a - b) ** 2 for a, b in zip(weights_fp16, dequantized)) / GROUP_SIZE print(f"