From 8bc08f451da1600e9fd3d26c41cea5c8e8764b0a Mon Sep 17 00:00:00 2001 From: Assistant builder Date: Thu, 11 Jun 2026 15:05:31 +0000 Subject: [PATCH] Prototype phase findings --- findings.md | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 findings.md diff --git a/findings.md b/findings.md new file mode 100644 index 0000000..8d49aae --- /dev/null +++ b/findings.md @@ -0,0 +1,141 @@ +# Prototype-phase findings + +Outcome of the two probes run on 2026-06-11 against the Pi at `192.168.50.115`. Both passed their spec criteria. This document captures what we learned, what to carry forward, and what to revisit when designing the main assistant. + +## What we proved + +- **Recording + playback works.** `python-sounddevice` + `soundfile` at 16 kHz mono PCM16 round-trips cleanly through the USB Speaker Phone on the Pi. Level meter responds to speech; playback is recognisable. +- **Wakeword detection works.** openwakeword's stock `alexa` model loads in ~0.6 s, detects reliably (user reports "runs perfectly"), false-positive rate is acceptable, CPU is well below saturation on a single core. + +## Environment facts to remember + +- Pi 4B, 4 GB, Debian 13 (trixie), Python 3.13.5, kernel `6.12.75+rpt-rpi-v8`. +- PipeWire is the audio server, but PortAudio sees only the raw ALSA hostapi — no PipeWire/Pulse hostapi. +- The USB Speaker Phone (Anhui LISTENAI, USB `f201:3220`) is exposed as ALSA card 3. **Native sample rate is 48 kHz only.** Raw `hw:3,0` rejects 16 kHz with `PaErrorCode -9997`. The ALSA `plughw:CARD=Phone,DEV=0` device exists and does the resampling. +- The same USB device is *both* mic and speaker. There is no separate output device. +- Pi `sudo` for user `pi` requires a password (not passwordless). + +## Code patterns to carry forward + +### 1. Force PortAudio through ALSA's `plug` plugin + +Set this **before importing `sounddevice`** in every test: + +```python +import os +os.environ.setdefault("PA_ALSA_PLUGHW", "1") +``` + +Best done in one place in the main assistant — e.g. an `assistant/audio.py` that sets the env var and re-exports `sounddevice`. The ordering is fragile if every entry point has to remember it. + +### 2. USB device discovery + +`find_usb_device()` is duplicated byte-identically in both tests: + +```python +def find_usb_device() -> int: + for idx, dev in enumerate(sd.query_devices()): + name = dev["name"].lower() + if "usb" in name and dev["max_input_channels"] >= 1: + return idx + print("USB audio device not found. Devices:", file=sys.stderr) + print(sd.query_devices(), file=sys.stderr) + sys.exit(1) +``` + +Lift to a shared module in the main assistant. + +### 3. openwakeword needs `--no-deps` on Python 3.13 + +`openwakeword==0.6.0` declares a hard dep on `tflite-runtime>=2.8.0,<3 ; platform_system == "Linux"`. There is no `tflite-runtime` wheel for Python 3.13 and pip refuses to install. + +Workaround: `bin/deploy` does two pip-install passes — a normal `requirements.txt` (sounddevice, numpy, onnxruntime, scipy, scikit-learn, tqdm, requests) and a `requirements-nodeps.txt` (openwakeword) installed with `--no-deps`. Keep this split for the main assistant until either the Pi drops to 3.12 or openwakeword stops hard-requiring tflite-runtime. + +### 4. `openwakeword.utils.download_models(target_directory=None)` crashes + +The function does `os.path.exists(target_directory)` with no None-handling. Call `download_models()` with no kwargs — the default is the package's `resources/models` dir, which is what you want. + +## Things to fix or redesign in the main assistant + +### A. Beep / TTS blocks the input stream + +In Test 2, `sd.play(beep) + sd.wait()` runs in the same thread that pulls from the InputStream. During the ~200 ms beep, the InputStream's ring buffer fills up; the next `stream.read()` returns stale audio and may flag input overflow. + +In the main assistant the same pattern would **drop the first ~200 ms of the user's command** right after the wakeword fires — the worst possible moment. + +Pattern for the main assistant: +- Callback-driven InputStream that pushes frames onto a `queue.Queue`. +- A consumer thread (or asyncio task) reads from the queue and feeds the wakeword model / Realtime stream. +- A separate, persistent OutputStream opened once at startup that the TTS / beep path pushes to. + +### B. The speaker is the mic + +The USB Speaker Phone hears its own output. TTS playback will re-trigger the wakeword on its own speech. + +Options: +- **Gate the detector during playback.** Set a flag while TTS is playing; skip `model.predict()` while it's set. Simple and reliable. +- **Use hardware AEC if the device has it.** The Anhui LISTENAI USB speakerphones often have on-board AEC — worth probing (try `arecord` while `aplay` is running, see if the input is clean). +- **Software AEC** (`webrtc-audio-processing`). Adds complexity; only if hardware AEC isn't enough. + +Recommendation: gate the detector during playback as the v1 behaviour. Revisit AEC only if barge-in becomes a requirement. + +### C. Custom wakeword (target: "hey assistant" or similar) + +Test 2 used the stock `alexa` model as a probe. The final wakeword needs to be trained. openwakeword has a Colab notebook that generates synthetic training data and trains a model in ~1 hour. Plan this as a separate task before the main assistant ships. + +### D. Single-core CPU budget + +The Pi 4 has 4 cores and the wakeword model uses well under 50% of one. The main assistant will add the Realtime WebSocket (small) plus opus encode/decode for the audio it sends and receives (non-trivial). Budget for: 1 core for audio I/O + wakeword, 1 core for Realtime / codec, headroom on the other two. + +### E. `bin/bootstrap` and the sudo password + +`bin/bootstrap` currently does `echo "$PI_PASS" | sudo -S` to authenticate sudo on the Pi. That puts the password into the remote process's `/proc//cmdline` briefly. Alternatives, in order of safety: + +1. **Enable passwordless sudo for `pi` on the Pi.** Cleanest. Requires explicit security decision. Add a single file `/etc/sudoers.d/010-pi-nopasswd` containing `pi ALL=(ALL) NOPASSWD: ALL`. +2. **Run bootstrap once manually, never script it.** Avoids the issue entirely. +3. **Switch to `expect` instead of sshpass + sudo -S.** More code, similar trust model. + +Pre-decide for the main assistant; the deploy/install scripts will hit this same wall. + +### F. No requirements lockfile + +For prototypes, pinning major versions in `requirements.txt` is fine. For the long-lived main assistant, switch to `pip-compile` (or `uv`) to produce a lockfile so the Pi doesn't drift over time. + +### G. `~/assistant/` layout vs. rsync `--delete` + +`bin/deploy` uses `rsync -az --delete` against `~/assistant/`. That's correct for a code dir but will nuke anything else placed there. For the main assistant, split the layout: + +- `~/assistant/code/` — rsync target with `--delete`. +- `~/assistant/state/` — recordings, logs, custom wakeword model, runtime data. Never rsynced. + +### H. systemd autostart + +Not in scope for prototypes. The main assistant needs a user systemd service so it survives reboots and restarts on crash. + +## Cosmetic / known-warnings + +- onnxruntime prints two warnings on startup: + ``` + Failed to detect devices under "/sys/class/drm/card0" + Failed to detect devices under "/sys/class/drm/card1" + ``` + It's probing for GPU vendor info; the Pi's vc4 framebuffer doesn't expose a `device/vendor` sysfs file. Harmless. Suppressible via `os.environ["ORT_DISABLE_GPU_DETECTION"] = "1"` (or similar onnxruntime log-level setting) if it bothers you. + +## Open questions for the next design session + +1. Wakeword phrase for the custom model — "hey assistant", "assistant", "computer", something else? +2. Conversation lifecycle — wakeword → open Realtime session → close when? On silence (VAD)? On user-spoken endword? On timeout? +3. Barge-in (user interrupting the assistant mid-reply) — in scope or not? +4. Where the OpenAI API key lives. (Env var? Pi-only file? Pulled from a secrets store at boot?) +5. Local-only fallback when the network is down — required behaviour or out of scope? +6. Multi-language? Or English only for v1? + +## Reference paths + +- Spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md` +- Plan: `docs/superpowers/plans/2026-06-11-voice-assistant-prototypes.md` +- Test 1: `tests/01-record-play/main.py` +- Test 2: `tests/02-wakeword/main.py` +- Deploy helper: `bin/deploy` +- Pi bootstrap: `bin/bootstrap` +- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)