Home Assistant Voice PE with a local LLM instead of Echo

What they built

The author ripped out every Echo and Nest Mini because each was "a hot mic with a privacy policy I did not write", then rebuilt voice control around Home Assistant's Voice Preview Edition.

Whisper, Piper and wake word run locally. An Ollama model on a separate GPU box handles fuzzy, conversational requests that the built-in intent engine can't.

Setup

• Home Assistant on an Intel N100 mini PC with 16 GB RAM
• Home Assistant Voice Preview Edition (~$60)
• Optional GPU box: used NVIDIA RTX 3060 12 GB running Ollama
• faster-whisper add-on (small-int8 on N100; medium if pointed at a GPU)
• Piper add-on, voice en_US-lessac-medium
• LLM: llama3.2:3b

How it works

1. Adopt the Voice PE in Home Assistant (USB-C, Wi-Fi, no app or account)
2. Install faster-whisper, Piper and optionally openWakeWord add-ons (Wyoming)
3. Build an Assist pipeline: built-in agent, faster-whisper STT, Piper TTS
4. Add the Ollama integration, enable Assist, and switch the pipeline's conversation agent to it

Results & numbers

• small-int8 Whisper on the N100 transcribes a short command in well under a second
• Built-in commands keep working with the internet unplugged
• With the GPU on CPU fallback, answers take ten seconds instead of one

Gotchas

• The tiny Whisper model mishears 'bedroom' as 'bathroom'
• Not checking that the model is on the GPU is the single most common mistake
• Without 'Do not make up device states', a 3B model will invent device states

“Pull your internet and try again. It still works. That moment is the whole reason I did this.”— Joule P. Kraft, jpk.io

Do it yourself

1. Adopt the Voice PE in Home Assistant
2. Install faster-whisper (small-int8) and Piper from the Add-on Store
3. Create an Assist pipeline with those engines and test with the internet unplugged
4. On a GPU box: install Ollama, pull llama3.2:3b, confirm `ollama ps` shows 100% GPU
5. Add the Ollama integration, enable Assist, set it as the conversation agent with a short system prompt

You are a voice assistant for a smart home.
Answer in one or two short sentences.
You can control devices that are exposed to you.
If you do not know, say so. Do not make up device states.