Build a Local AI Voice Assistant with Whisper, Ollama, and Piper: A Complete 2026 Tutorial

Voice assistants like Siri, Alexa, and Google Assistant all share one thing: they send your voice to a server. Every command, every question, every whispered request at 2am gets processed on someone else’s hardware. For privacy-conscious users, developers building on-device products, or anyone who just wants full control over their AI stack, that is a dealbreaker.

The good news is that the tools for building a fully local voice assistant have matured significantly. OpenAI’s Whisper runs speech recognition on consumer hardware. Ollama lets you run large language models locally. Piper provides fast, natural-sounding text-to-speech. Together, they form a pipeline that can listen, think, and speak — all without an internet connection.

This tutorial walks through building that pipeline from scratch. By the end, you will have a working voice assistant that runs entirely on your machine.

What You Will Build

The system has three components:

Speech-to-Text (STT): OpenAI Whisper converts your spoken words into text
Language Model (LLM): Ollama runs a local language model that processes the text and generates a response
Text-to-Speech (TTS): Piper converts the text response back into spoken audio

The flow is simple: you speak into a microphone, Whisper transcribes your speech, the LLM generates a response, and Piper speaks it back to you. Everything happens locally. No API keys, no cloud services, no data leaving your machine.

Why Build Local?

The privacy argument is the most obvious one. Cloud-based voice assistants store your recordings, and while companies claim they do not use the data for training, the fine print often tells a different story. Running everything locally means your voice commands never leave your network.

But there are practical reasons too. Latency is one. A cloud-based voice assistant has to upload your audio, process it on a remote server, and stream the response back. Even with fast internet, that round trip adds 500ms to 2 seconds of latency. A local pipeline on decent hardware can cut that to under 1 second.

Cost is another factor. Running Whisper, a 3B LLM, and Piper costs nothing beyond the electricity your computer already uses. Compare that to API costs: Whisper via OpenAI’s API charges $0.006 per minute of audio. GPT-4o-mini costs $0.15 per million input tokens. For a voice assistant you use daily, those costs add up. The local approach has zero marginal cost after the initial setup.

There is also the reliability angle. Cloud services go down. APIs hit rate limits. Network connections drop. A local voice assistant works regardless of your internet status. If your Wi-Fi goes out, you can still ask it to set a timer, look up a recipe, or draft an email.

Prerequisites

You need a machine with:

Python 3.10 or later
At least 8GB of RAM (16GB recommended for larger models)
A working microphone
macOS, Linux, or Windows (WSL2)

The tutorial assumes basic command-line familiarity but does not require AI or ML experience.

Step 1: Install Ollama

Ollama is the simplest way to run language models locally. It handles model downloading, quantization, and GPU acceleration automatically. Think of it as Docker for LLMs — you pull a model, run it, and it just works.

Install Ollama from the official site:

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Or download from https://ollama.com/download for Windows

Once installed, pull a model. For a voice assistant, you want something fast and capable. Llama 3.2 3B is a good balance between speed and intelligence. The 3B parameter count means it runs comfortably on machines with 8GB of RAM while still handling conversational queries well.

ollama pull llama3.2:3b

Test that it works:

ollama run llama3.2:3b "Say hello in one sentence."

If you get a response, Ollama is ready. The first inference will be slow as the model loads into memory, but subsequent calls will be fast — typically under a second on modern hardware.

Ollama also supports streaming responses, which matters for a voice assistant. Instead of waiting for the full response before speaking, you could pipe each sentence to Piper as it arrives. We will cover that optimization later in the tutorial.

Step 2: Install Whisper

Whisper is OpenAI’s open-source speech recognition model. We will use the faster-whisper implementation, which runs at 4x speed with lower memory usage compared to the original.

pip install faster-whisper

Download a model. The base model is fast and accurate enough for most use cases:

from faster_whisper import WhisperModel

model = WhisperModel("base", device="cpu", compute_type="int8")
segments, info = model.transcribe("test_audio.wav")
for segment in segments:
    print(segment.text)

If you have an NVIDIA GPU, use device="cuda" and compute_type="float16" for significantly faster transcription. On Apple Silicon, device="cpu" with compute_type="int8" already runs in near real-time.

Step 3: Install Piper

Piper is a fast, local text-to-speech engine. It produces natural-sounding speech without requiring a GPU.

pip install piper-tts

Download a voice model. Piper offers dozens of voices in multiple languages:

# List available voices at https://rhasspy.github.io/piper-samples/
# Download a voice (example: US English female)
pip install piper-tts
python -m piper_download --voice en_US-lessac-medium

Test the TTS:

echo "Hello, I am your local voice assistant." | piper --output_file test.wav

Play the resulting WAV file to verify the voice sounds natural.

Step 4: Build the Pipeline

Now combine the three components into a working assistant. Here is a Python script that ties everything together:

import os
import tempfile
import subprocess
import sounddevice as sd
import numpy as np
from faster_whisper import WhisperModel
import ollama

# Configuration
SAMPLE_RATE = 16000
DURATION = 5  # seconds to record
WHISPER_MODEL = "base"
LLM_MODEL = "llama3.2:3b"
PIPER_VOICE = "en_US-lessac-medium"

# Initialize Whisper
whisper = WhisperModel(WHISPER_MODEL, device="cpu", compute_type="int8")

def record_audio():
    """Record audio from the microphone."""
    print("Listening... (speak now)")
    audio = sd.rec(int(DURATION * SAMPLE_RATE), samplerate=SAMPLE_RATE, channels=1, dtype='float32')
    sd.wait()
    print("Done listening.")
    return audio.flatten()

def transcribe(audio):
    """Convert audio to text using Whisper."""
    with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
        import wave
        with wave.open(f.name, 'w') as wav:
            wav.setnchannels(1)
            wav.setsampwidth(2)
            wav.setframerate(SAMPLE_RATE)
            wav.writeframes((audio * 32767).astype(np.int16).tobytes())
        
        segments, _ = whisper.transcribe(f.name)
        text = " ".join(segment.text for segment in segments)
    
    os.unlink(f.name)
    return text

def ask_llm(prompt):
    """Send prompt to local LLM and get response."""
    response = ollama.chat(model=LLM_MODEL, messages=[
        {"role": "system", "content": "You are a helpful voice assistant. Keep responses concise and conversational."},
        {"role": "user", "content": prompt}
    ])
    return response["message"]["content"]

def speak(text):
    """Convert text to speech using Piper."""
    with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
        subprocess.run(
            ["piper", "--model", PIPER_VOICE, "--output_file", f.name],
            input=text.encode(),
            capture_output=True
        )
        subprocess.run(["aplay", f.name])  # Linux; use 'afplay' on macOS
        os.unlink(f.name)

def main():
    print("Local Voice Assistant ready. Press Ctrl+C to exit.")
    try:
        while True:
            audio = record_audio()
            text = transcribe(audio)
            if not text.strip():
                continue
            print(f"You said: {text}")
            
            response = ask_llm(text)
            print(f"Assistant: {response}")
            
            speak(response)
    except KeyboardInterrupt:
        print("\nGoodbye!")

if __name__ == "__main__":
    main()

Install the additional dependency:

pip install sounddevice numpy

Run the assistant:

python assistant.py

Step 5: Fine-Tuning and Optimization

The basic pipeline works, but several adjustments improve the experience.

Voice Activity Detection

The current script records for a fixed 5 seconds regardless of whether you are speaking. Add voice activity detection (VAD) to stop recording when you stop talking:

# Install: pip install webrtcvad
import webrtcvad

def detect_silence(audio, sample_rate=16000, silence_threshold=0.01):
    """Simple energy-based silence detection."""
    frame_size = int(sample_rate * 0.03)  # 30ms frames
    is_silent = True
    for i in range(0, len(audio) - frame_size, frame_size):
        frame = audio[i:i+frame_size]
        energy = np.sqrt(np.mean(frame**2))
        if energy > silence_threshold:
            is_silent = False
            break
    return is_silent

Conversation Context

The basic script treats each query independently. For a more natural experience, maintain conversation history:

conversation_history = []

def ask_llm(prompt):
    global conversation_history
    conversation_history.append({"role": "user", "content": prompt})
    
    # Keep last 10 exchanges for context
    messages = [
        {"role": "system", "content": "You are a helpful voice assistant. Keep responses concise."},
        *conversation_history[-10:]
    ]
    
    response = ollama.chat(model=LLM_MODEL, messages=messages)
    reply = response["message"]["content"]
    conversation_history.append({"role": "assistant", "content": reply})
    return reply

Wake Word Detection

For a hands-free experience, add a wake word like “hey assistant.” You can use pvporcupine (free tier available) or a simple keyword spotter:

pip install pvporcupine

This makes the assistant only respond when addressed, rather than recording continuously.

Response Length Control

LLMs sometimes generate lengthy responses that take forever to speak. Add a system prompt constraint:

system_prompt = """You are a helpful voice assistant. Rules:
- Keep responses under 3 sentences
- Be conversational, not formal
- If the answer is simple, give a simple answer
- Never use markdown, bullet points, or lists in spoken responses"""

Hardware Considerations

Performance depends heavily on your hardware:

CPU only (8GB RAM): Whisper base model + Llama 3.2 3B works well. Response time: 2-4 seconds.
CPU only (16GB RAM): Can run Whisper medium + Llama 3.2 8B. Slower but more capable.
NVIDIA GPU (8GB+ VRAM): Whisper large-v3 + Llama 3.2 8B with near-instant response.
Apple Silicon (M1/M2/M3): Excellent performance with Core ML acceleration. Whisper base runs in real-time.

If you have limited RAM, use quantized models. Ollama automatically uses 4-bit quantization for most models, which reduces memory usage significantly. A 3B parameter model in 4-bit quantization uses roughly 2GB of RAM, leaving plenty of headroom for Whisper and Piper to run simultaneously.

For storage, plan on about 2-3GB total: roughly 2GB for the LLM, 150MB for Whisper base, and 50-100MB for a Piper voice model. These are one-time downloads — after the initial pull, the models live on your disk and load from local storage.

Troubleshooting

Common issues and fixes:

Whisper is slow: Use compute_type="int8" on CPU, or switch to a smaller model (tiny instead of base).

No audio input: Check your microphone permissions. On macOS, go to System Settings > Privacy > Microphone. On Linux, verify with arecord -l.

Piper sounds robotic: Try a different voice model. The medium quality voices sound significantly better than low quality ones.

Ollama hangs: The first inference after loading a model is slow. Subsequent calls are fast. If it consistently hangs, check available RAM.

Going Further

This tutorial builds a basic pipeline. To extend it:

Add tool use: Let the LLM call functions (weather, calendar, web search) through Ollama’s function calling support. This transforms the assistant from a conversational tool into something that can actually perform tasks.
Multi-language: Use Whisper’s multilingual models and Piper voices in other languages. Whisper supports 99 languages out of the box, and Piper has voices for over 30 languages. Building a bilingual or multilingual assistant is mostly a matter of selecting the right voice model.
GUI frontend: Build a simple web interface with Gradio or Streamlit for visual feedback. A waveform display showing when the assistant is listening, a transcript of the conversation, and a status indicator for which component is active all make the experience more polished.
Edge deployment: Run the pipeline on a Raspberry Pi 5 for a dedicated hardware device. The Pi 5’s 8GB RAM is enough for Whisper base and Llama 3.2 3B, and the result is a standalone voice assistant that sits on your desk.
Streaming TTS: Instead of waiting for the full LLM response, split it into sentences and pipe each one to Piper as it completes. This cuts perceived latency in half because the assistant starts speaking before it has finished thinking.
Custom wake words: Train a custom wake word using Picovoice’s Porcupine or Snowboy. This lets you call your assistant whatever you want instead of relying on a preset keyword.

The local AI voice assistant space is moving fast. A year ago, running these models on consumer hardware was impractical. Today, it is straightforward. The tools will only get better.

Model Choices and Tradeoffs

The model you pick for each component affects the experience significantly. Here is what we have found works:

For Whisper, the base model strikes the best balance for real-time use. It handles clear speech well, processes audio faster than real-time on modern CPUs, and rarely makes mistakes on common phrases. The small model adds about 30% more accuracy on accented or mumbled speech but runs slower. The medium and large models are overkill for a voice assistant — they excel at transcription accuracy but are too slow for interactive use without a GPU.

For the LLM, the sweet spot depends on your patience. Llama 3.2 3B responds in under a second on most hardware, making it ideal for a voice assistant where every extra second of silence feels awkward. If you have a GPU and do not mind a 2-3 second wait, the 8B model gives noticeably better answers, especially for multi-step questions or creative tasks.

For Piper, the medium quality voices are the default recommendation. They sound natural without requiring significant processing power. The high quality voices sound slightly better but are noticeably slower to generate. For a voice assistant, speed matters more than marginal quality improvements — you want the response to start speaking as soon as possible after the LLM finishes.