# Prototype-phase findings

Outcome of the two probes run on 2026-06-11 against the Pi at `192.168.50.115`. Both passed their spec criteria. This document captures what we learned, what to carry forward, and what to revisit when designing the main assistant.

## What we proved

- **Recording + playback works.** `python-sounddevice` + `soundfile` at 16 kHz mono PCM16 round-trips cleanly through the USB Speaker Phone on the Pi. Level meter responds to speech; playback is recognisable.
- **Wakeword detection works.** openwakeword's stock `alexa` model loads in ~0.6 s, detects reliably (user reports "runs perfectly"), false-positive rate is acceptable, CPU is well below saturation on a single core.

## Environment facts to remember

- Pi 4B, 4 GB, Debian 13 (trixie), Python 3.13.5, kernel `6.12.75+rpt-rpi-v8`.
- PipeWire is the audio server, but PortAudio sees only the raw ALSA hostapi — no PipeWire/Pulse hostapi.
- The USB Speaker Phone (Anhui LISTENAI, USB `f201:3220`) is exposed as ALSA card 3. **Native sample rate is 48 kHz only.** Raw `hw:3,0` rejects 16 kHz with `PaErrorCode -9997`. The ALSA `plughw:CARD=Phone,DEV=0` device exists and does the resampling.
- The same USB device is *both* mic and speaker. There is no separate output device.
- Pi `sudo` for user `pi` requires a password (not passwordless).

## Code patterns to carry forward

### 1. Force PortAudio through ALSA's `plug` plugin

Set this **before importing `sounddevice`** in every test:

```python
import os
os.environ.setdefault("PA_ALSA_PLUGHW", "1")
```

Best done in one place in the main assistant — e.g. an `assistant/audio.py` that sets the env var and re-exports `sounddevice`. The ordering is fragile if every entry point has to remember it.

### 2. USB device discovery

`find_usb_device()` is duplicated byte-identically in both tests:

```python
def find_usb_device() -> int:
    for idx, dev in enumerate(sd.query_devices()):
        name = dev["name"].lower()
        if "usb" in name and dev["max_input_channels"] >= 1:
            return idx
    print("USB audio device not found. Devices:", file=sys.stderr)
    print(sd.query_devices(), file=sys.stderr)
    sys.exit(1)
```

Lift to a shared module in the main assistant.

### 3. openwakeword needs `--no-deps` on Python 3.13

`openwakeword==0.6.0` declares a hard dep on `tflite-runtime>=2.8.0,<3 ; platform_system == "Linux"`. There is no `tflite-runtime` wheel for Python 3.13 and pip refuses to install.

Workaround: `bin/deploy` does two pip-install passes — a normal `requirements.txt` (sounddevice, numpy, onnxruntime, scipy, scikit-learn, tqdm, requests) and a `requirements-nodeps.txt` (openwakeword) installed with `--no-deps`. Keep this split for the main assistant until either the Pi drops to 3.12 or openwakeword stops hard-requiring tflite-runtime.

### 4. `openwakeword.utils.download_models(target_directory=None)` crashes

The function does `os.path.exists(target_directory)` with no None-handling. Call `download_models()` with no kwargs — the default is the package's `resources/models` dir, which is what you want.

## Things to fix or redesign in the main assistant

### A. Beep / TTS blocks the input stream

In Test 2, `sd.play(beep) + sd.wait()` runs in the same thread that pulls from the InputStream. During the ~200 ms beep, the InputStream's ring buffer fills up; the next `stream.read()` returns stale audio and may flag input overflow.

In the main assistant the same pattern would **drop the first ~200 ms of the user's command** right after the wakeword fires — the worst possible moment.

Pattern for the main assistant:
- Callback-driven InputStream that pushes frames onto a `queue.Queue`.
- A consumer thread (or asyncio task) reads from the queue and feeds the wakeword model / Realtime stream.
- A separate, persistent OutputStream opened once at startup that the TTS / beep path pushes to.

### B. The speaker is the mic

The USB Speaker Phone hears its own output. TTS playback will re-trigger the wakeword on its own speech.

Options:
- **Gate the detector during playback.** Set a flag while TTS is playing; skip `model.predict()` while it's set. Simple and reliable.
- **Use hardware AEC if the device has it.** The Anhui LISTENAI USB speakerphones often have on-board AEC — worth probing (try `arecord` while `aplay` is running, see if the input is clean).
- **Software AEC** (`webrtc-audio-processing`). Adds complexity; only if hardware AEC isn't enough.

Recommendation: gate the detector during playback as the v1 behaviour. Revisit AEC only if barge-in becomes a requirement.

### C. Custom wakeword (target: "hey assistant" or similar)

Test 2 used the stock `alexa` model as a probe. The final wakeword needs to be trained. openwakeword has a Colab notebook that generates synthetic training data and trains a model in ~1 hour. Plan this as a separate task before the main assistant ships.

### D. Single-core CPU budget

The Pi 4 has 4 cores and the wakeword model uses well under 50% of one. The main assistant will add the Realtime WebSocket (small) plus opus encode/decode for the audio it sends and receives (non-trivial). Budget for: 1 core for audio I/O + wakeword, 1 core for Realtime / codec, headroom on the other two.

### E. `bin/bootstrap` and the sudo password

`bin/bootstrap` currently does `echo "$PI_PASS" | sudo -S` to authenticate sudo on the Pi. That puts the password into the remote process's `/proc/<pid>/cmdline` briefly. Alternatives, in order of safety:

1. **Enable passwordless sudo for `pi` on the Pi.** Cleanest. Requires explicit security decision. Add a single file `/etc/sudoers.d/010-pi-nopasswd` containing `pi ALL=(ALL) NOPASSWD: ALL`.
2. **Run bootstrap once manually, never script it.** Avoids the issue entirely.
3. **Switch to `expect` instead of sshpass + sudo -S.** More code, similar trust model.

Pre-decide for the main assistant; the deploy/install scripts will hit this same wall.

### F. No requirements lockfile

For prototypes, pinning major versions in `requirements.txt` is fine. For the long-lived main assistant, switch to `pip-compile` (or `uv`) to produce a lockfile so the Pi doesn't drift over time.

### G. `~/assistant/` layout vs. rsync `--delete`

`bin/deploy` uses `rsync -az --delete` against `~/assistant/`. That's correct for a code dir but will nuke anything else placed there. For the main assistant, split the layout:

- `~/assistant/code/` — rsync target with `--delete`.
- `~/assistant/state/` — recordings, logs, custom wakeword model, runtime data. Never rsynced.

### H. systemd autostart

Not in scope for prototypes. The main assistant needs a user systemd service so it survives reboots and restarts on crash.

## Cosmetic / known-warnings

- onnxruntime prints two warnings on startup:
  ```
  Failed to detect devices under "/sys/class/drm/card0"
  Failed to detect devices under "/sys/class/drm/card1"
  ```
  It's probing for GPU vendor info; the Pi's vc4 framebuffer doesn't expose a `device/vendor` sysfs file. Harmless. Suppressible via `os.environ["ORT_DISABLE_GPU_DETECTION"] = "1"` (or similar onnxruntime log-level setting) if it bothers you.

## Open questions for the next design session

1. Wakeword phrase for the custom model — "hey assistant", "assistant", "computer", something else?
2. Conversation lifecycle — wakeword → open Realtime session → close when? On silence (VAD)? On user-spoken endword? On timeout?
3. Barge-in (user interrupting the assistant mid-reply) — in scope or not?
4. Where the OpenAI API key lives. (Env var? Pi-only file? Pulled from a secrets store at boot?)
5. Local-only fallback when the network is down — required behaviour or out of scope?
6. Multi-language? Or English only for v1?

## C# probe outcome (2026-06-12 — Test 1 ported to C#)

The Python Test 1 (record 5 s → write WAV → play back) was re-implemented in C# / .NET 9 to validate that the Pi client can be written in the same language as the backend. The full probe lives at `tests/01-record-play-cs/` with the deploy script at `bin/probe-cs`. All three pass criteria from the original spec held on hardware.

### What we proved

- **PortAudioSharp2 1.0.6** is a usable .NET binding for PortAudio. Conventional API surface (`PortAudio.Initialize`, `Stream` ctor taking input/output `StreamParameters`, callback delegate with `ref StreamCallbackTimeInfo`) — matched our assumptions, no adjustments needed.
- The NuGet package **bundles `libportaudio.so` for linux-arm64** in its runtime folder, so a self-contained publish doesn't need to rely on the Pi's system `libportaudio2`. (Both are present; the bundled one wins.)
- A single `dotnet publish -c Release -r linux-arm64 --self-contained` produces a ~70 MB directory that runs as-is on the Pi. **No .NET install on the Pi.** Deploy is `dotnet publish → scp -r → ssh ./Probe`, wrapped in `bin/probe-cs`.
- The `PA_ALSA_PLUGHW=1` trick from the original findings is still required in C# — but with a real twist (see next section).
- Recording, RMS metering, WAV writing, and playback all behave identically to the Python prototype. Pass criteria 1 (recognisable playback), 2 (meter responsive), 3 (no over/underflow) all met.

### Key gotcha — .NET env vars do NOT propagate to libc `getenv()` on Linux

`Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1")` updates .NET's process-table mirror but does **not** call libc `setenv`. libportaudio reads its config via `getenv("PA_ALSA_PLUGHW")` and sees nothing. ALSA then opens the USB Speaker Phone as `hw:3,0` (native 48 kHz only), and the 16 kHz stream fails with `paInvalidSampleRate`.

Python's `os.environ[x] = y` calls `setenv` synchronously, which is why the Python probe never tripped on this.

**Fix:** P/Invoke libc `setenv` directly, alongside the managed call:

```csharp
[DllImport("libc", EntryPoint = "setenv")]
static extern int setenv(string name, string value, int overwrite);

Environment.SetEnvironmentVariable("PA_ALSA_PLUGHW", "1"); // for .NET-side readers
setenv("PA_ALSA_PLUGHW", "1", 1);                           // for the C runtime view PortAudio reads
```

**Implication for future C# probes and the eventual Pi client:** any env knob a C library reads (PortAudio, onnxruntime, any P/Invoked native) must be set via libc `setenv`, not just `Environment.SetEnvironmentVariable`. Bake a small `Libc.setenv` helper into the eventual client.

### Small .NET ergonomics

- `<AllowUnsafeBlocks>true</AllowUnsafeBlocks>` is required in the csproj for the callback's `Buffer.MemoryCopy` over `short*`.
- `ImplicitUsings` pulls in `System.IO.Stream`, which collides with `PortAudioSharp.Stream`. Disambiguate with `using Stream = PortAudioSharp.Stream;`.
- Top-level statements don't auto-pick up the project's own namespace. Add `using RecordPlayProbe;` (or whatever the project's namespace is) so the entry-point file can reach internal helpers.

### Open questions deferred to Test 2

- **No equivalent to openwakeword exists in .NET.** The choices are (a) port the openwakeword inference pipeline to C# using `Microsoft.ML.OnnxRuntime` (mel extractor → Google speech embedder → keyword classifier, three ONNX files), (b) use Picovoice Porcupine's .NET SDK (commercial, key-gated), (c) Vosk + keyword spotting (heavier). Decision deferred to the Test 2 brainstorm.
- The `Libc.setenv` workaround will likely be needed again for any onnxruntime env knob on the Pi.

### Reference paths (C# probe)

- Spec: `docs/superpowers/specs/2026-06-12-record-play-csharp-probe-design.md`
- Plan: `docs/superpowers/plans/2026-06-12-record-play-csharp-probe.md`
- Code: `tests/01-record-play-cs/`
- Deploy: `bin/probe-cs`

## Reference paths

- Spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`
- Plan: `docs/superpowers/plans/2026-06-11-voice-assistant-prototypes.md`
- Test 1: `tests/01-record-play/main.py`
- Test 2: `tests/02-wakeword/main.py`
- Deploy helper: `bin/deploy`
- Pi bootstrap: `bin/bootstrap`
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
