Spec: voice assistant prototype phase
This commit is contained in:
@@ -0,0 +1,132 @@
|
||||
# Voice Assistant — prototype phase design
|
||||
|
||||
**Date:** 2026-06-11
|
||||
**Status:** approved
|
||||
**Scope:** two throwaway test projects that de-risk audio I/O and wakeword detection on the target device before main development of the OpenAI Realtime voice assistant begins.
|
||||
|
||||
## Context
|
||||
|
||||
The end goal is a voice assistant on a Raspberry Pi 4B that streams microphone audio to the OpenAI Realtime API and plays back the model's speech. Before building that, we need to confirm that the two foundational pieces work on this exact hardware:
|
||||
|
||||
1. **Voice recording and playback** — can we capture clean mic audio and play it back through the speaker at the sample rate and format the Realtime API expects?
|
||||
2. **Wakeword detection** — can we run a wakeword model locally on the Pi 4 in real time, accurately enough to gate the Realtime session?
|
||||
|
||||
Both tests are scoped as standalone scripts. They are not meant to evolve into the final assistant — they exist to answer yes/no questions. The final assistant will be designed separately after both probes return "yes."
|
||||
|
||||
## Target device
|
||||
|
||||
- Raspberry Pi 4B, 4 GB RAM, aarch64
|
||||
- Debian 13 (trixie), kernel `6.12.75+rpt-rpi-v8`
|
||||
- Python 3.13.5 (`/usr/bin/python3`)
|
||||
- Audio server: PipeWire (active, user session)
|
||||
- Audio device: ALSA card 3, `USB Speaker Phone` (Anhui LISTENAI, USB ID `f201:3220`) — combined microphone and speaker on a single USB device
|
||||
- Reached over SSH from this sandbox at `pi@192.168.50.115` (see `CLAUDE.md`)
|
||||
|
||||
The USB Speaker Phone is the only mic + speaker on the device. The HDMI outputs and bcm2835 headphone jack are present but unused for these tests.
|
||||
|
||||
## Wakeword choice
|
||||
|
||||
- Engine: **openwakeword** (Apache 2.0, runs on CPU, no API key)
|
||||
- Model for the test: stock `alexa` — chosen because it ships with the library and lets us validate the engine end-to-end without training. The final assistant will use a custom-trained "assistant" (or similar) model; training that is a separate task that happens after this prototype phase passes.
|
||||
|
||||
## Audio I/O library
|
||||
|
||||
`python-sounddevice` (PortAudio bindings) for both tests:
|
||||
|
||||
- One consistent stack across record, play, and wakeword feeding.
|
||||
- Streaming callback API that matches openwakeword's chunked input requirement.
|
||||
- Plays nicely with PipeWire's ALSA shim — no PipeWire-specific code needed.
|
||||
- Same library will be reused in the main assistant for the OpenAI Realtime stream.
|
||||
|
||||
`soundfile` is used in test 1 for WAV round-tripping.
|
||||
|
||||
`pyaudio` and `arecord`/`aplay` subprocess were considered and rejected: `pyaudio` has no advantage and a worse install story on Debian 13; the subprocess approach is awkward for the wakeword feeding loop.
|
||||
|
||||
## Repository layout
|
||||
|
||||
```
|
||||
Assistant/
|
||||
├── CLAUDE.md
|
||||
├── bin/
|
||||
│ └── deploy
|
||||
├── tests/
|
||||
│ ├── 01-record-play/
|
||||
│ │ ├── main.py
|
||||
│ │ └── requirements.txt
|
||||
│ └── 02-wakeword/
|
||||
│ ├── main.py
|
||||
│ └── requirements.txt
|
||||
└── docs/superpowers/specs/
|
||||
└── 2026-06-11-voice-assistant-prototypes-design.md # this file
|
||||
```
|
||||
|
||||
Source of truth lives in `/app/data/home/Assistant` (in this sandbox). Code is rsynced to the Pi to run. One shared venv on the Pi at `~/assistant/.venv` is created on first deploy; each test's `requirements.txt` is installed into it.
|
||||
|
||||
### `bin/deploy`
|
||||
|
||||
A small shell helper. Usage: `bin/deploy <test-dir>`, e.g. `bin/deploy tests/02-wakeword`.
|
||||
|
||||
Responsibilities:
|
||||
1. `rsync` the project tree to `pi@192.168.50.115:~/assistant/`.
|
||||
2. Ensure `~/assistant/.venv` exists (create with `python3 -m venv` if not).
|
||||
3. `pip install -r <test-dir>/requirements.txt` inside the venv.
|
||||
4. SSH in and run `<test-dir>/main.py` with the venv's Python, attaching to the terminal so the user can interact and see output live.
|
||||
|
||||
The script must not echo `$GIT_KEY` / `$GIT_USER`, and pipes ssh/rsync stderr through `sed` to redact them in case of error output. Auth uses `sshpass -p assistant`.
|
||||
|
||||
## Test 1 — voice recording and playback
|
||||
|
||||
`tests/01-record-play/main.py` does, in order:
|
||||
|
||||
1. Open `sounddevice.InputStream` on the USB Speaker Phone, 16 kHz, mono, `int16`.
|
||||
2. Record 5 seconds while printing a one-line, updating-in-place RMS meter so the operator can see the mic is producing signal.
|
||||
3. Write the buffer to `out.wav` via `soundfile.write` (16 kHz mono PCM16).
|
||||
4. Open `sounddevice.OutputStream` on the same device and play `out.wav` back, blocking until finished.
|
||||
|
||||
### Pass criteria
|
||||
|
||||
- Playback is recognisable as what was said into the mic.
|
||||
- Level meter visibly responds to speech and to silence.
|
||||
- No clipping, dropouts, or sustained underrun warnings from PortAudio.
|
||||
|
||||
### Error handling
|
||||
|
||||
- If the USB Speaker Phone is not found (no ALSA card whose name contains `USB`), print the output of `sounddevice.query_devices()` and exit with a non-zero status. Suggest the user check `arecord -l`.
|
||||
- Any PortAudio error during streaming aborts with the raw error message — no retry.
|
||||
|
||||
## Test 2 — wakeword
|
||||
|
||||
`tests/02-wakeword/main.py` does:
|
||||
|
||||
1. On startup, load openwakeword's stock `alexa` model. If the model files are not yet cached locally, openwakeword's downloader fetches them. Print the model load time.
|
||||
2. Open `sounddevice.InputStream` on the USB Speaker Phone, 16 kHz, mono, `int16`, with a frame size of **1280 samples (80 ms)** — openwakeword's expected chunk size.
|
||||
3. In the input callback, call `model.predict(frame)` and read the `alexa` score from the returned dict.
|
||||
4. When the score crosses **0.5**, print `DETECTED alexa score=<value> t=<elapsed-since-start>` and play a short (~200 ms) sine-wave beep through the same device. Keep listening afterwards. Apply a 1 s cooldown so a single trigger event doesn't fire repeatedly.
|
||||
5. `Ctrl-C` exits cleanly (close the stream, exit 0).
|
||||
|
||||
### Pass criteria
|
||||
|
||||
- Saying "alexa" at normal speaking volume from ~1 m away triggers a detection within ~1 s, reliably (≥ 8 out of 10 attempts).
|
||||
- During roughly a minute of unrelated speech (reading aloud, conversation), false positives stay under ~1 per minute.
|
||||
- CPU usage stays well below saturating a single core (`htop` shows the Python process < 50% on one core).
|
||||
|
||||
### Error handling
|
||||
|
||||
- Audio device missing: same behaviour as test 1.
|
||||
- Model download failure (no network on first run): print the underlying exception and exit non-zero. Suggest re-running once the Pi has internet.
|
||||
|
||||
## Out of scope
|
||||
|
||||
The following are explicitly **not** part of this phase:
|
||||
|
||||
- OpenAI Realtime API wiring, session management, streaming audio to and from the model.
|
||||
- Voice activity detection, barge-in, conversation turn-taking.
|
||||
- A custom-trained wakeword model for the final wake phrase.
|
||||
- A systemd service / autostart for the assistant.
|
||||
- Any production-grade error handling, logging, or observability.
|
||||
|
||||
These are intentionally deferred. Each will get its own spec after both tests above pass.
|
||||
|
||||
## Success of this phase
|
||||
|
||||
Both tests pass their criteria when run on the Pi. After that, we open a new design doc for the main assistant.
|
||||
Reference in New Issue
Block a user