Offline LLM on a Raspberry Pi 5 for classification and extraction

What they built

Stefano Noferi runs a quantised model on an 8 GB Raspberry Pi 5 with no network. The point isn't throughput. The text never crosses a network boundary, which removes data-transfer obligations by construction.

The Pi is treated as the place for the smallest model that solves a precise task: classify an email, extract fields, route a request or summarise a paragraph.

Setup

• Raspberry Pi 5, 8 GB, 64-bit Raspberry Pi OS
• llama.cpp built with NEON/dotprod (or Ollama for prototyping)
• Phi-3 mini 3.8B at Q4_K_M (~2.3 GB)

Results & numbers

• 1-2B models at Q4: tens of tokens/s
• Phi-3 mini 3.8B Q4_K_M: ~4-6 tokens/s
• 7-8B at Q4: under 2 tokens/s, not interactive
• llama.cpp from source is ~10-20% faster than Ollama

Gotchas

• Memory, not compute, is the limit: the practical ceiling is ~4B at Q4
• Q4 quantization degrades quality; measure it on your task, especially for multi-step reasoning
• Validate on real data, not published benchmarks

“Keeping inference on the device cuts the transfer at the root, not as a mitigation measure but as a physical fact.”— Stefano Noferi

Do it yourself

1. Flash 64-bit Raspberry Pi OS on an 8 GB Pi 5
2. Build llama.cpp natively (or install Ollama to prototype)
3. Download Phi-3 mini Q4_K_M GGUF
4. Design narrow prompts (classify, extract, route) rather than chat
5. Validate accuracy on your own data