Fully local Home Assistant voice assistant on a GPU

What they built

Joe Karlsson replaced every cloud AI piece of Home Assistant's voice pipeline with local services: GPU-accelerated faster-whisper for speech-to-text, Piper for text-to-speech and Ollama as the conversation agent.

The components talk over Home Assistant's Wyoming protocol and can live on different machines; his do. The goal was simple: no audio leaves the network.

Setup

• GPU host: NVIDIA Quadro RTX 4000
• STT: faster-whisper (small, later medium-int8) as a Wyoming service
• TTS: Piper, voice en_US-libritts-high
• LLM: llama3.2:3b at first, qwen3:8b since summer 2026 (thinking disabled)
• Three pipelines: Local Voice (preferred), Ollama Local on CPU, and HA Cloud as last-resort fallback

How it works

1. Wake word detected, mic audio streamed to faster-whisper on the GPU host
2. Text goes to Home Assistant; with prefer_local_intents, built-in intents handle simple device commands directly
3. Anything else (questions, multi-step requests) goes to the Ollama model
4. The reply is spoken back through Piper

Results & numbers

• Round trip from wake word to audio: 2-3 seconds on GPU
• Medium-int8 Whisper transcribes a short command in about a third of a second
• July 2026 test: qwen3:8b took ~1.3 s per call and got every request right with HA's real prompt and 171 tool definitions
• Earlier tool-calling reliability: qwen3:4b ~70%, llama3.2:3b ~80% on simple commands

Gotchas

• qwen3:4b with think mode narrated its internal reasoning through the speaker for about five minutes
• Local models were inconsistent at tool calling on complex multi-device requests
• Conversation-trigger automations clash with the LLM pipeline; use HA's built-in voice intents for device control instead

“Intents for device control. Automations only for custom queries that need live sensor data. LLM as a last resort for everything else.”— Joe Karlsson

Do it yourself

1. Install Ollama on a GPU machine and pull a tool-capable model (the author now uses qwen3:8b, thinking off)
2. In Home Assistant add the faster-whisper and Piper Wyoming services
3. Add the Ollama integration pointing at http://<gpu-host>:11434
4. Create a voice pipeline and enable prefer_local_intents
5. Use built-in intents for device control; keep automations for custom queries only
6. Optionally keep a second, cloud pipeline as a fallback

name: Local Voice
stt_engine: stt.faster_whisper_gpu
stt_language: en
tts_engine: tts.piper
tts_voice: en_US-libritts-high
conversation_engine: conversation.ollama_conversation
prefer_local_intents: true