142 lines
7.9 KiB
Markdown
142 lines
7.9 KiB
Markdown
# Prototype-phase findings
|
|
|
|
Outcome of the two probes run on 2026-06-11 against the Pi at `192.168.50.115`. Both passed their spec criteria. This document captures what we learned, what to carry forward, and what to revisit when designing the main assistant.
|
|
|
|
## What we proved
|
|
|
|
- **Recording + playback works.** `python-sounddevice` + `soundfile` at 16 kHz mono PCM16 round-trips cleanly through the USB Speaker Phone on the Pi. Level meter responds to speech; playback is recognisable.
|
|
- **Wakeword detection works.** openwakeword's stock `alexa` model loads in ~0.6 s, detects reliably (user reports "runs perfectly"), false-positive rate is acceptable, CPU is well below saturation on a single core.
|
|
|
|
## Environment facts to remember
|
|
|
|
- Pi 4B, 4 GB, Debian 13 (trixie), Python 3.13.5, kernel `6.12.75+rpt-rpi-v8`.
|
|
- PipeWire is the audio server, but PortAudio sees only the raw ALSA hostapi — no PipeWire/Pulse hostapi.
|
|
- The USB Speaker Phone (Anhui LISTENAI, USB `f201:3220`) is exposed as ALSA card 3. **Native sample rate is 48 kHz only.** Raw `hw:3,0` rejects 16 kHz with `PaErrorCode -9997`. The ALSA `plughw:CARD=Phone,DEV=0` device exists and does the resampling.
|
|
- The same USB device is *both* mic and speaker. There is no separate output device.
|
|
- Pi `sudo` for user `pi` requires a password (not passwordless).
|
|
|
|
## Code patterns to carry forward
|
|
|
|
### 1. Force PortAudio through ALSA's `plug` plugin
|
|
|
|
Set this **before importing `sounddevice`** in every test:
|
|
|
|
```python
|
|
import os
|
|
os.environ.setdefault("PA_ALSA_PLUGHW", "1")
|
|
```
|
|
|
|
Best done in one place in the main assistant — e.g. an `assistant/audio.py` that sets the env var and re-exports `sounddevice`. The ordering is fragile if every entry point has to remember it.
|
|
|
|
### 2. USB device discovery
|
|
|
|
`find_usb_device()` is duplicated byte-identically in both tests:
|
|
|
|
```python
|
|
def find_usb_device() -> int:
|
|
for idx, dev in enumerate(sd.query_devices()):
|
|
name = dev["name"].lower()
|
|
if "usb" in name and dev["max_input_channels"] >= 1:
|
|
return idx
|
|
print("USB audio device not found. Devices:", file=sys.stderr)
|
|
print(sd.query_devices(), file=sys.stderr)
|
|
sys.exit(1)
|
|
```
|
|
|
|
Lift to a shared module in the main assistant.
|
|
|
|
### 3. openwakeword needs `--no-deps` on Python 3.13
|
|
|
|
`openwakeword==0.6.0` declares a hard dep on `tflite-runtime>=2.8.0,<3 ; platform_system == "Linux"`. There is no `tflite-runtime` wheel for Python 3.13 and pip refuses to install.
|
|
|
|
Workaround: `bin/deploy` does two pip-install passes — a normal `requirements.txt` (sounddevice, numpy, onnxruntime, scipy, scikit-learn, tqdm, requests) and a `requirements-nodeps.txt` (openwakeword) installed with `--no-deps`. Keep this split for the main assistant until either the Pi drops to 3.12 or openwakeword stops hard-requiring tflite-runtime.
|
|
|
|
### 4. `openwakeword.utils.download_models(target_directory=None)` crashes
|
|
|
|
The function does `os.path.exists(target_directory)` with no None-handling. Call `download_models()` with no kwargs — the default is the package's `resources/models` dir, which is what you want.
|
|
|
|
## Things to fix or redesign in the main assistant
|
|
|
|
### A. Beep / TTS blocks the input stream
|
|
|
|
In Test 2, `sd.play(beep) + sd.wait()` runs in the same thread that pulls from the InputStream. During the ~200 ms beep, the InputStream's ring buffer fills up; the next `stream.read()` returns stale audio and may flag input overflow.
|
|
|
|
In the main assistant the same pattern would **drop the first ~200 ms of the user's command** right after the wakeword fires — the worst possible moment.
|
|
|
|
Pattern for the main assistant:
|
|
- Callback-driven InputStream that pushes frames onto a `queue.Queue`.
|
|
- A consumer thread (or asyncio task) reads from the queue and feeds the wakeword model / Realtime stream.
|
|
- A separate, persistent OutputStream opened once at startup that the TTS / beep path pushes to.
|
|
|
|
### B. The speaker is the mic
|
|
|
|
The USB Speaker Phone hears its own output. TTS playback will re-trigger the wakeword on its own speech.
|
|
|
|
Options:
|
|
- **Gate the detector during playback.** Set a flag while TTS is playing; skip `model.predict()` while it's set. Simple and reliable.
|
|
- **Use hardware AEC if the device has it.** The Anhui LISTENAI USB speakerphones often have on-board AEC — worth probing (try `arecord` while `aplay` is running, see if the input is clean).
|
|
- **Software AEC** (`webrtc-audio-processing`). Adds complexity; only if hardware AEC isn't enough.
|
|
|
|
Recommendation: gate the detector during playback as the v1 behaviour. Revisit AEC only if barge-in becomes a requirement.
|
|
|
|
### C. Custom wakeword (target: "hey assistant" or similar)
|
|
|
|
Test 2 used the stock `alexa` model as a probe. The final wakeword needs to be trained. openwakeword has a Colab notebook that generates synthetic training data and trains a model in ~1 hour. Plan this as a separate task before the main assistant ships.
|
|
|
|
### D. Single-core CPU budget
|
|
|
|
The Pi 4 has 4 cores and the wakeword model uses well under 50% of one. The main assistant will add the Realtime WebSocket (small) plus opus encode/decode for the audio it sends and receives (non-trivial). Budget for: 1 core for audio I/O + wakeword, 1 core for Realtime / codec, headroom on the other two.
|
|
|
|
### E. `bin/bootstrap` and the sudo password
|
|
|
|
`bin/bootstrap` currently does `echo "$PI_PASS" | sudo -S` to authenticate sudo on the Pi. That puts the password into the remote process's `/proc/<pid>/cmdline` briefly. Alternatives, in order of safety:
|
|
|
|
1. **Enable passwordless sudo for `pi` on the Pi.** Cleanest. Requires explicit security decision. Add a single file `/etc/sudoers.d/010-pi-nopasswd` containing `pi ALL=(ALL) NOPASSWD: ALL`.
|
|
2. **Run bootstrap once manually, never script it.** Avoids the issue entirely.
|
|
3. **Switch to `expect` instead of sshpass + sudo -S.** More code, similar trust model.
|
|
|
|
Pre-decide for the main assistant; the deploy/install scripts will hit this same wall.
|
|
|
|
### F. No requirements lockfile
|
|
|
|
For prototypes, pinning major versions in `requirements.txt` is fine. For the long-lived main assistant, switch to `pip-compile` (or `uv`) to produce a lockfile so the Pi doesn't drift over time.
|
|
|
|
### G. `~/assistant/` layout vs. rsync `--delete`
|
|
|
|
`bin/deploy` uses `rsync -az --delete` against `~/assistant/`. That's correct for a code dir but will nuke anything else placed there. For the main assistant, split the layout:
|
|
|
|
- `~/assistant/code/` — rsync target with `--delete`.
|
|
- `~/assistant/state/` — recordings, logs, custom wakeword model, runtime data. Never rsynced.
|
|
|
|
### H. systemd autostart
|
|
|
|
Not in scope for prototypes. The main assistant needs a user systemd service so it survives reboots and restarts on crash.
|
|
|
|
## Cosmetic / known-warnings
|
|
|
|
- onnxruntime prints two warnings on startup:
|
|
```
|
|
Failed to detect devices under "/sys/class/drm/card0"
|
|
Failed to detect devices under "/sys/class/drm/card1"
|
|
```
|
|
It's probing for GPU vendor info; the Pi's vc4 framebuffer doesn't expose a `device/vendor` sysfs file. Harmless. Suppressible via `os.environ["ORT_DISABLE_GPU_DETECTION"] = "1"` (or similar onnxruntime log-level setting) if it bothers you.
|
|
|
|
## Open questions for the next design session
|
|
|
|
1. Wakeword phrase for the custom model — "hey assistant", "assistant", "computer", something else?
|
|
2. Conversation lifecycle — wakeword → open Realtime session → close when? On silence (VAD)? On user-spoken endword? On timeout?
|
|
3. Barge-in (user interrupting the assistant mid-reply) — in scope or not?
|
|
4. Where the OpenAI API key lives. (Env var? Pi-only file? Pulled from a secrets store at boot?)
|
|
5. Local-only fallback when the network is down — required behaviour or out of scope?
|
|
6. Multi-language? Or English only for v1?
|
|
|
|
## Reference paths
|
|
|
|
- Spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`
|
|
- Plan: `docs/superpowers/plans/2026-06-11-voice-assistant-prototypes.md`
|
|
- Test 1: `tests/01-record-play/main.py`
|
|
- Test 2: `tests/02-wakeword/main.py`
|
|
- Deploy helper: `bin/deploy`
|
|
- Pi bootstrap: `bin/bootstrap`
|
|
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
|