Gemma 4 Local Setup and Deployment
Gemma 4 runs locally via Ollama, llama.cpp, LM Studio, vLLM, and others. Use Q8 quant for quality. Chat template differs from Gemma 3. Architecture uses dual RoPE and sliding/global attention.
Gemma 4 is available on Hugging Face, Ollama, and Kaggle. It is compatible with llama.cpp, LM Studio, vLLM, MLX, NVIDIA NIM, Unsloth, and other inference frameworks. Key setup notes: - Only instruction-tuned (-it) variants have GGUF quantizations for llama.cpp and LM Studio. - Recommended quantization: Q8 for best all-around quality. Ollama defaults to Q4. - Tools may need updating to support the Gemma 4 chat template, which uses standard system/user/assistant roles (unlike Gemma 3). - Do not feed prior thought blocks back into conversation history in multi-turn chats. Architecture details: Built on the same research stack as Gemini 3. Uses alternating local sliding-window attention (512-1024 tokens) and global full-context attention. Employs dual RoPE — standard for sliding layers, proportional for global layers — enabling 256K context. Vocabulary size is 262,144. Audio encoder uses a USM-style conformer (same as Gemma-3n). Designed for quantization-friendliness and broad library compatibility.