llama.cpp

**llama.cpp** is a C++ inference runtime for large language models, created by Bulgarian engineer **Georgi Gerganov** (also known as ggerganov) starting in March 2023. It has become the reference implementation for running LLMs locally on consumer hardware and is the inference engine inside most popular local-LLM user applications. ## Origin In March 2023, Meta released LLaMA (Large Language Model Meta AI) weights to researchers; the weights leaked publicly within days. Gerganov, working as a solo developer, wrote a minimal C++ inference implementation that could run the 7B-parameter model on an Apple M1 MacBook Pro at usable speed — which at the time felt miraculous because the expected hardware was a datacenter GPU cluster. The initial release (llama.cpp) was **~500 lines of C++**. Within weeks, community contributions added quantization, GPU support, multiple model architectures, and a REST API. The project exploded. ## What it does - **Inference only** — no training. Takes a quantized model file and runs generation. - **Pure C++** — no Python dependency. Compiles to a single binary. - **Extensive hardware support**: CPU (SIMD-optimized for AVX2/AVX512/NEON/etc), Apple Metal (M-series), CUDA (Nvidia), ROCm (AMD), Vulkan (cross-vendor GPU), SYCL (Intel GPU), OpenCL, Ascend (Huawei NPU). - **Quantization**: K-quant and iQuant (2-8 bits per weight), dramatically reducing memory requirements. - **Model architecture support**: dozens of architectures (LLaMA 1/2/3/4, Mistral, Qwen, Gemma, Phi, Falcon, GPT-2/J/NeoX, MPT, Starcoder, CodeLlama, DeepSeek, GLM, MiniMax, and many more). - **OpenAI-compatible server**: `llama-server` provides a REST API matching OpenAI's Chat Completions format. - **Tool calling, function calling, embeddings**: supported. - **Multimodal**: vision (LLaVA family), audio (Whisper.cpp sibling project). ## GGUF format **GGUF** (GGML Universal Format) is the file format for quantized LLMs used by llama.cpp. It is a binary format combining model weights, architecture metadata, tokenizer, and chat template into one file. GGUF files are typically named like `llama-3-70b-q4_k_m.gguf`, indicating the model and quantization type. GGUF replaced the earlier GGML and GGJT formats as llama.cpp matured. Most open-weight model releases in 2024-2026 are quickly converted to GGUF by community contributors (notably TheBloke, Unsloth, MaziyarPanahi, and bartowski). ## Quantization tiers Typical GGUF quant levels (higher = more bits per weight = better quality, more memory): - **Q8_0**: 8 bits/weight, ~near-FP16 quality, 50% memory savings. - **Q6_K**: ~6.5 bits/weight, excellent quality. - **Q5_K_M**: ~5.7 bits/weight, 'sweet spot' for many users. - **Q4_K_M**: ~4.5 bits/weight, standard recommendation, 25% of FP16 memory. - **Q4_0**: 4 bits/weight, slight quality loss. - **Q3_K_M**: ~3.5 bits/weight, noticeable quality loss, only for memory-constrained. - **Q2_K**: 2.5 bits/weight, significant quality loss, rarely usable. For a 70B model at FP16 (~140GB), Q4_K_M runs at ~40GB — fits in a 48GB MacBook or a 24GB GPU + system RAM. ## Ecosystem built on llama.cpp - **LM Studio** — consumer desktop app with GGUF model browser. - **Ollama** — CLI-focused local-LLM runner with model library. - **Jan** — open-source alternative to ChatGPT desktop client. - **Text Generation WebUI** (oobabooga) — Gradio-based web UI. - **KoboldCpp** — llama.cpp fork with creative-writing focus. - **AnythingLLM** — Tim Carambat's desktop local-LLM framework. - **GPT4All** — consumer chat app. - Embedded in many apps' 'run a local LLM' feature. ## Significance llama.cpp's impact on the AI ecosystem is hard to overstate: 1. **Democratized local inference**: before llama.cpp, running a 7B-parameter model required a data-center GPU. After: a MacBook runs 70B-parameter models. 2. **Accelerated open-weight adoption**: every open-weight model release now includes 'GGUF available within hours.' 3. **Independence from cloud providers**: developers can build on open models without API dependencies. 4. **Reference for quantization research**: the K-quant method has influenced academic quantization literature. 5. **Performance benchmark**: new model releases are routinely benchmarked in llama.cpp-based runtime configurations. ## Development model llama.cpp is open source (MIT license). Gerganov continues as lead maintainer; the project has hundreds of contributors and is one of the most-starred AI repositories on GitHub (~100K+ stars as of 2026). Gerganov founded **ggml.ai** as a related commercial entity (inference as a service, consulting). The core llama.cpp project remains fully open. ## Related - GLM 5.1 Open-Weight Model, MiniMax M2.7 — models typically run via llama.cpp. - Open Source vs Open Weight Debate — llama.cpp is the practical substrate for the open-weight ecosystem. - Cosmopolitan Libc — Justine Tunney contributed Cosmopolitan-compiled llamafile builds (llama.cpp in a one-binary-any-OS package).