# Prototype-phase findings

Outcome of the two probes run on 2026-06-11 against the Pi at `192.168.50.115`. Both passed their spec criteria. This document captures what we learned, what to carry forward, and what to revisit when designing the main assistant.

## What we proved

- **Recording + playback works.** `python-sounddevice` + `soundfile` at 16 kHz mono PCM16 round-trips cleanly through the USB Speaker Phone on the Pi. Level meter responds to speech; playback is recognisable.
- **Wakeword detection works.** openwakeword's stock `alexa` model loads in ~0.6 s, detects reliably (user reports "runs perfectly"), false-positive rate is acceptable, CPU is well below saturation on a single core.

## Environment facts to remember

- Pi 4B, 4 GB, Debian 13 (trixie), Python 3.13.5, kernel `6.12.75+rpt-rpi-v8`.
- PipeWire is the audio server, but PortAudio sees only the raw ALSA hostapi — no PipeWire/Pulse hostapi.
- The USB Speaker Phone (Anhui LISTENAI, USB `f201:3220`) is exposed as ALSA card 3. **Native sample rate is 48 kHz only.** Raw `hw:3,0` rejects 16 kHz with `PaErrorCode -9997`. The ALSA `plughw:CARD=Phone,DEV=0` device exists and does the resampling.
- The same USB device is *both* mic and speaker. There is no separate output device.
- Pi `sudo` for user `pi` requires a password (not passwordless).

## Code patterns to carry forward

### 1. Force PortAudio through ALSA's `plug` plugin

Set this **before importing `sounddevice`** in every test:

```python
import os
os.environ.setdefault("PA_ALSA_PLUGHW", "1")
```

Best done in one place in the main assistant — e.g. an `assistant/audio.py` that sets the env var and re-exports `sounddevice`. The ordering is fragile if every entry point has to remember it.

### 2. USB device discovery

`find_usb_device()` is duplicated byte-identically in both tests:

```python
def find_usb_device() -> int:
    for idx, dev in enumerate(sd.query_devices()):
        name = dev["name"].lower()
        if "usb" in name and dev["max_input_channels"] >= 1:
            return idx
    print("USB audio device not found. Devices:", file=sys.stderr)
    print(sd.query_devices(), file=sys.stderr)
    sys.exit(1)
```

Lift to a shared module in the main assistant.

### 3. openwakeword needs `--no-deps` on Python 3.13

`openwakeword==0.6.0` declares a hard dep on `tflite-runtime>=2.8.0,<3 ; platform_system == "Linux"`. There is no `tflite-runtime` wheel for Python 3.13 and pip refuses to install.

Workaround: `bin/deploy` does two pip-install passes — a normal `requirements.txt` (sounddevice, numpy, onnxruntime, scipy, scikit-learn, tqdm, requests) and a `requirements-nodeps.txt` (openwakeword) installed with `--no-deps`. Keep this split for the main assistant until either the Pi drops to 3.12 or openwakeword stops hard-requiring tflite-runtime.

### 4. `openwakeword.utils.download_models(target_directory=None)` crashes

The function does `os.path.exists(target_directory)` with no None-handling. Call `download_models()` with no kwargs — the default is the package's `resources/models` dir, which is what you want.

## Things to fix or redesign in the main assistant

### A. Beep / TTS blocks the input stream

In Test 2, `sd.play(beep) + sd.wait()` runs in the same thread that pulls from the InputStream. During the ~200 ms beep, the InputStream's ring buffer fills up; the next `stream.read()` returns stale audio and may flag input overflow.

In the main assistant the same pattern would **drop the first ~200 ms of the user's command** right after the wakeword fires — the worst possible moment.

Pattern for the main assistant:
- Callback-driven InputStream that pushes frames onto a `queue.Queue`.
- A consumer thread (or asyncio task) reads from the queue and feeds the wakeword model / Realtime stream.
- A separate, persistent OutputStream opened once at startup that the TTS / beep path pushes to.

### B. The speaker is the mic

The USB Speaker Phone hears its own output. TTS playback will re-trigger the wakeword on its own speech.

Options:
- **Gate the detector during playback.** Set a flag while TTS is playing; skip `model.predict()` while it's set. Simple and reliable.
- **Use hardware AEC if the device has it.** The Anhui LISTENAI USB speakerphones often have on-board AEC — worth probing (try `arecord` while `aplay` is running, see if the input is clean).
- **Software AEC** (`webrtc-audio-processing`). Adds complexity; only if hardware AEC isn't enough.

Recommendation: gate the detector during playback as the v1 behaviour. Revisit AEC only if barge-in becomes a requirement.

### C. Custom wakeword (target: "hey assistant" or similar)

Test 2 used the stock `alexa` model as a probe. The final wakeword needs to be trained. openwakeword has a Colab notebook that generates synthetic training data and trains a model in ~1 hour. Plan this as a separate task before the main assistant ships.

### D. Single-core CPU budget

The Pi 4 has 4 cores and the wakeword model uses well under 50% of one. The main assistant will add the Realtime WebSocket (small) plus opus encode/decode for the audio it sends and receives (non-trivial). Budget for: 1 core for audio I/O + wakeword, 1 core for Realtime / codec, headroom on the other two.

### E. `bin/bootstrap` and the sudo password

`bin/bootstrap` currently does `echo "$PI_PASS" | sudo -S` to authenticate sudo on the Pi. That puts the password into the remote process's `/proc/<pid>/cmdline` briefly. Alternatives, in order of safety:

1. **Enable passwordless sudo for `pi` on the Pi.** Cleanest. Requires explicit security decision. Add a single file `/etc/sudoers.d/010-pi-nopasswd` containing `pi ALL=(ALL) NOPASSWD: ALL`.
2. **Run bootstrap once manually, never script it.** Avoids the issue entirely.
3. **Switch to `expect` instead of sshpass + sudo -S.** More code, similar trust model.

Pre-decide for the main assistant; the deploy/install scripts will hit this same wall.

### F. No requirements lockfile

For prototypes, pinning major versions in `requirements.txt` is fine. For the long-lived main assistant, switch to `pip-compile` (or `uv`) to produce a lockfile so the Pi doesn't drift over time.

### G. `~/assistant/` layout vs. rsync `--delete`

`bin/deploy` uses `rsync -az --delete` against `~/assistant/`. That's correct for a code dir but will nuke anything else placed there. For the main assistant, split the layout:

- `~/assistant/code/` — rsync target with `--delete`.
- `~/assistant/state/` — recordings, logs, custom wakeword model, runtime data. Never rsynced.

### H. systemd autostart

Not in scope for prototypes. The main assistant needs a user systemd service so it survives reboots and restarts on crash.

## Cosmetic / known-warnings

- onnxruntime prints two warnings on startup:
  ```
  Failed to detect devices under "/sys/class/drm/card0"
  Failed to detect devices under "/sys/class/drm/card1"
  ```
  It's probing for GPU vendor info; the Pi's vc4 framebuffer doesn't expose a `device/vendor` sysfs file. Harmless. Suppressible via `os.environ["ORT_DISABLE_GPU_DETECTION"] = "1"` (or similar onnxruntime log-level setting) if it bothers you.

## Open questions for the next design session

1. Wakeword phrase for the custom model — "hey assistant", "assistant", "computer", something else?
2. Conversation lifecycle — wakeword → open Realtime session → close when? On silence (VAD)? On user-spoken endword? On timeout?
3. Barge-in (user interrupting the assistant mid-reply) — in scope or not?
4. Where the OpenAI API key lives. (Env var? Pi-only file? Pulled from a secrets store at boot?)
5. Local-only fallback when the network is down — required behaviour or out of scope?
6. Multi-language? Or English only for v1?

## Reference paths

- Spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`
- Plan: `docs/superpowers/plans/2026-06-11-voice-assistant-prototypes.md`
- Test 1: `tests/01-record-play/main.py`
- Test 2: `tests/02-wakeword/main.py`
- Deploy helper: `bin/deploy`
- Pi bootstrap: `bin/bootstrap`
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)