Local AI use cases with llama.cpp
llama.cpp is the open-source C/C++ inference engine behind much of local AI. It runs quantized GGUF models on CPUs and GPUs and includes llama-server, an OpenAI-compatible API.
Rebel BitesHome RAG assistant for embedded-systems docs on Proxmox
Ollama, llama.cpp, Open WebUI, BGE-M3 and SearXNG in a GPU-passthrough VM. Moving from 6 GB to 16 GB of VRAM changed which models could run.
Replacing Google Nest Minis with local Assist on llama.cpp
After a year of Nest Minis getting dumber, a household moved to fully local Assist. Higher-quant GGUF models from Hugging Face are what finally fixed tool calling.
Obsidian vault Q&A with Mini-RAG and llama.cpp
An Obsidian plugin that chats with a note or folder, using separate local llama.cpp servers for chat and embeddings (Ollama works too).
Security camera descriptions and search with Frigate GenAI
Frigate sends camera frames to a local vision model (qwen3-vl via Ollama or llama.cpp) for searchable event descriptions, review summaries and chat.
Offline chat on desktop or Android via Termux with Prefrontal
A privacy-first chat UI that works with Ollama on desktop or llama.cpp anywhere, including Android through Termux. No telemetry, no subscriptions.
Offline LLM on a Raspberry Pi 5 for classification and extraction
Phi-3 mini at Q4_K_M runs at 4-6 tokens per second on an 8 GB Pi 5 with no network at all. Under GDPR, data that never leaves the board isn't transferred anywhere.