Work AI
I'm building a household voice assistant around a locally hosted model on a Mac Studio, usable from any device on my Tailscale network.
The project brings together local inference, custom MCP servers, built-in live tools, a notification system, and a push-to-talk web app.
Web app on any device on the tailnet, sending audio over a WebSocket
mlx-whisper large-v3-turbo on Apple silicon
Qwen3.8-27B or Flash-Next on a Mac Studio, served by oMLX
Custom MCP servers and built-in tools
Kokoro text-to-speech, running locally
I switch between two Qwen3.8 models on oMLX. Only one fits in memory at a time, and the same model also runs a local coding agent, so the choice is a trade between speed and room for other work.
Leaves enough memory for cloud coding agents and test devices to run beside it.
About twice as fast per request and higher on the assistant eval, but it needs most of the Mac's memory.
Each candidate is scored on an eval of the real assistant and measured for speed and memory use. Earlier versions ran Qwen3.6 35B-A3B and, first, Gemma 4 12B on an RTX 5070 Ti.
The assistant's model runs locally on a Mac Studio, sharing its 64 GB of unified memory with coding agents.
MCP servers connect the assistant to lights and music.
Built-in tools look up live grocery offers from local Danish supermarkets and handle the calendar, shared lists, reminders, memory, and web search.
A push-to-talk web app with Whisper speech recognition and Kokoro speech, reached from any device over Tailscale. Notifications go through a self-hosted ntfy server.
Moving work out of the model and into a tool.
The smaller local models couldn't extract the right offers from the supermarket pages without overflowing their context.
SolutionThe extraction became a tool. A script fetches the offers, keeps the ones that are valid today, sorts them by unit price, and returns a short list with only the fields the model needs.
A grocery question now triggers the lookup automatically before the model answers.
The new version widens this into a rule for any changing fact, such as prices, weather or opening hours: if the model answers one from memory without calling a tool, the reply is held back and the model is asked to look it up.
Voice sets a hard latency budget. When I ask for the lights to be turned off, I don't want to wait 30 seconds for it to happen.
Everything in the system prompt is processed before the first reply. Keeping it lean, and only offering tools whose backend is actually up, keeps answers fast and keeps the model on track.
How fast the first word arrives depends on how quickly the model reads the prompt, not just how fast it generates. The fixed part of the prompt is cached when the assistant starts, and anything that changes, like the clock, goes in the user's message, so each turn only reads what's new.
Whisper mishears names like Occhio. A vocabulary file corrects those words after recognition, so a command works the first time. Putting them in Whisper's prompt instead made its English recognition worse.