From 99a9950ceeff0dcf3287044d6f4abe15f893a92d Mon Sep 17 00:00:00 2001 From: Assistant builder Date: Thu, 11 Jun 2026 15:48:10 +0000 Subject: [PATCH] Smart-assistant main-build design spec Captures the architecture for the post-prototype build: Pi Python client + ASP.NET backend (with embedded React UI) on Coolify, SQLite, multi-turn Realtime sessions with end_session/get_current_time/set_volume tools, pairing-code device onboarding, and idempotent install/update flow. --- .../2026-06-11-smart-assistant-design.md | 516 ++++++++++++++++++ 1 file changed, 516 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-11-smart-assistant-design.md diff --git a/docs/superpowers/specs/2026-06-11-smart-assistant-design.md b/docs/superpowers/specs/2026-06-11-smart-assistant-design.md new file mode 100644 index 0000000..5f4e4b6 --- /dev/null +++ b/docs/superpowers/specs/2026-06-11-smart-assistant-design.md @@ -0,0 +1,516 @@ +# Smart Assistant — design + +**Date:** 2026-06-11 +**Status:** approved (brainstorming complete) +**Scope:** main build of the voice assistant, replacing the throwaway prototypes in `tests/`. Two deliverables: a Python client on the Raspberry Pi, and an ASP.NET backend (with embedded React management UI) deployed to Coolify. Brings up everything needed to wake on a wakeword, hold a multi-turn conversation through the OpenAI Realtime API, persist a text history per device, run a small built-in tool registry, and let users manage their devices through a browser. + +## Context + +The prototype phase (`docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`, results in `findings.md`) proved that recording/playback and openwakeword both work on the target Pi. The main assistant is built on top of those findings. + +Key carry-forwards from the prototypes: + +- USB Speaker Phone (ALSA card 3, Anhui LISTENAI) is the only mic/speaker. 48 kHz native; `PA_ALSA_PLUGHW=1` set before importing `sounddevice` lets us open at any rate. +- `python-sounddevice` + `numpy` are the audio stack. +- `openwakeword` runs at ~0.6 s load, well under 50% of one core; the `alexa` stock model is reused for v1 (custom-trained wakeword is a separate follow-up task). +- `openwakeword==0.6.0` needs `--no-deps` install on Python 3.13 because its `tflite-runtime` pin has no wheel; the prototype's two-pass install pattern is kept. +- Speaker hears its own output → detector must be gated while playing audio, and uplink to OpenAI must be muted while the assistant is speaking. +- TTS/beep must not block the input stream — callback-driven InputStream + persistent OutputStream is the pattern. + +## Goals + +1. A fresh Raspberry Pi can be paired to a user account by running one curl-piped install command and entering a short code. +2. The Pi wakes on the wakeword, opens a Realtime conversation, supports multi-turn until the user says "bye" or goes idle, and plays optional `wake.wav` / `sleep.wav` cues. +3. A signed-in user sees only their own devices and conversations in the management UI. +4. An admin user can manage system-wide defaults and other users. +5. Per-device configuration: name, system prompt override, voice, model, idle timeout, enabled-tools toggles. Changes take effect on the next session. +6. Text-only conversation history per device, viewable in the UI. +7. Built-in C# tool registry with `end_session`, `get_current_time`, `set_volume` shipped in v1. +8. Backend deployed to Coolify as one container with one domain; deploys triggered fire-and-forget via the Coolify v4 API. +9. The install script is the update mechanism: re-running it upgrades the client in place without re-pairing. + +## Non-goals (v1) + +- Long-term memory across sessions. +- MCP-based external tools (only built-in C# tools). +- Audio recordings of conversations (text only). +- Live conversation tailing or audio meters in the UI. +- Per-device upload of `wake.wav`/`sleep.wav` via UI (filesystem only — v1.1). +- Custom-trained wakeword (uses the stock `alexa` model for v1). +- Barge-in (interrupting the assistant mid-reply). +- Multi-language (English only). +- Offline / local-only fallback when the network is down. +- OIDC sign-in (local ASP.NET Identity only). +- `.deb` packaging for the client. +- Load / performance testing. + +## Architecture + +### Topology + +``` +┌─────────────────────┐ pair code / token ┌──────────────────────────────────────────┐ +│ Browser (admin UI) │ ──────HTTPS (REST)────────▶ │ │ +│ React SPA │ │ ASP.NET backend (Coolify container) │ +└─────────────────────┘ │ │ + │ ┌────────────┐ ┌────────────────────┐ │ +┌─────────────────────┐ WSS audio + control │ │ REST/UI │ │ Device hub │ │ +│ Pi client (Python) │ ◀───────────────────────────▶│ │ + Identity│ │ (per-device WS) │ │ +│ wakeword + audio │ │ └────────────┘ └─────────┬──────────┘ │ +└─────────────────────┘ │ │ │ + │ ┌────────────┐ ┌────────▼──────────┐ │ + │ │ Tools │ │ Realtime relay │ │ + │ │ registry │◀─┤ (per-session WS │ │ + │ └────────────┘ │ upstream) │ │ + │ └─────────┬──────────┘ │ + │ SQLite (named volume) │ │ + └────────────────────────────┼────────────┘ + │ WSS + ▼ + OpenAI Realtime API +``` + +One container; React SPA served by the same ASP.NET process. SQLite file lives on a Coolify named volume. The OpenAI API key is held server-side; clients never see it. + +### Why backend relay (not direct Pi ↔ OpenAI) + +- One API key, server-side only. +- All user-side and assistant-side transcripts land in our database directly off the upstream WS events (no separate STT pass). +- Tool definitions and changes don't require a client redeploy. +- Backend can push audio to a Pi on its own (future feature) over the same WS. + +### Backend + +Single ASP.NET 9 project (`backend/`) organised into modules: + +``` +backend/ +├── Program.cs # composition root, EF migrate, route + hub mapping +├── Auth/ # ASP.NET Identity, roles (Admin, User), invite flow +├── Devices/ # device CRUD, pairing-code issuance, token mgmt +├── DeviceHub/ # WebSocket handler at /device, per-device session state +├── Realtime/ # RealtimeRelay: per-conversation upstream WS to OpenAI +├── Tools/ # ITool interface, registry, built-in tools +├── Conversations/ # persistence of turns, transcripts, tool calls +├── Install/ # serves install.sh and client.tar.gz +├── Data/ # EF Core DbContext, migrations, SQLite +└── wwwroot/ # built React SPA (Vite output) + client.tar.gz +``` + +Key services (DI): + +- `DeviceRegistry` (singleton) — in-memory map `deviceId → ActiveDevice`, tracks online state and last frame timestamp. +- `RealtimeSessionFactory` (singleton) — given a `DeviceContext`, opens an upstream WSS to OpenAI and returns a `RealtimeSession` instance bound to one conversation. +- `ToolRegistry` (singleton) — static list of `ITool`s registered at startup, filterable by per-device enabled set. +- `ConversationLog` (scoped) — append-only writer for turns; flushes per event so a crash mid-conversation still preserves what was said. +- `OpenAIKeyProvider` (singleton) — reads `OPENAI_API_KEY` from env. + +### Pi client + +Python package (`client/`), deployed to `~/assistant/code/` on the Pi: + +``` +client/ +├── main.py # entry: sets PA_ALSA_PLUGHW BEFORE importing sounddevice +├── audio.py # USB device discovery, persistent in/out streams +├── wakeword.py # openwakeword wrapper + gating flag +├── session.py # backend WS client: binary audio + JSON control +├── playback.py # queue-fed playback, wake/sleep wav hooks +├── state.py # state machine +├── config.py # loads ~/assistant/state/config.json +├── pair.py # one-shot: prompt for code, hit /api/pair, write config.json +└── log.py # structured logging to journald +``` + +State persists at `~/assistant/state/`: +- `config.json` (mode 0600) — `device_id`, `device_token`, `backend_ws` +- `wake.wav` / `sleep.wav` — optional cues, played if present +- (room for future per-device assets; never rsynced/overwritten by deploys) + +`~/assistant/code/` is the only directory that the install script replaces on update; `~/assistant/state/` is untouched. + +### State machine + +``` + wakeword fires session_started + ┌─────────────────────┐ ┌──────────────────────┐ + │ ▼ │ ▼ +[IDLE] ─────────► [WAKE_PENDING] ─────► [LISTENING] ───────► [ASSISTANT_SPEAKING] + ▲ │ ▲ │ + │ │ │ assistant done │ + │ └───┴──────────────────┘ + │ session_ended + └─────────────────────────────── (idle timeout OR end_session tool) + plays sleep.wav, re-enables wakeword +``` + +- **IDLE** — wakeword detector enabled, no upstream audio. Persistent backend WS open with heartbeats. +- **WAKE_PENDING** — wakeword fired; play `wake.wav` (non-blocking); send `wake`; wait for `session_started`. Wakeword disabled. +- **LISTENING** — mic frames stream to backend; server-VAD on OpenAI decides turn boundaries; we don't compute local VAD. +- **ASSISTANT_SPEAKING** — uplink muted; inbound frames flow to playback queue. +- On `assistant_done` from backend → LISTENING again (uplink unmuted). +- On `session_ended` from backend → wait for current audio to drain, play `sleep.wav`, send `playback_done(for=sleep)` → IDLE. + +### Audio plumbing + +- **Wire format**: PCM16 little-endian, 24 kHz mono, exactly 1920 samples (3840 bytes, 80 ms) per binary frame. Matches OpenAI Realtime's default `pcm16`. No transcoding in the backend. +- **Capture**: single InputStream, callback-driven, blocksize 1920, sample rate 24 000 Hz. Callback only enqueues — never blocks. +- **Playback**: single OutputStream opened at startup, persistent. Playback worker pulls from an `asyncio.Queue` and writes — never `sd.play()/sd.wait()` (which would block the InputStream the way the prototype did). +- **Wakeword feed**: each captured 1920-sample frame is downsampled to 1280 samples (80 ms at 16 kHz) via `scipy.signal.resample_poly(frame, up=2, down=3)` and fed to `model.predict()`. +- `PA_ALSA_PLUGHW=1` is set at the very top of `main.py` before any `sounddevice` import. + +### Threading model + +- Main thread runs an asyncio loop owning the backend WS. +- PortAudio invokes the InputStream callback on its own thread; the callback writes mic frames to a bounded `queue.Queue`. On overflow, oldest frame is dropped and a warning is logged. +- A bridge thread reads `queue.Queue`, runs wakeword.predict() when state is IDLE/WAKE_PENDING, and forwards bytes to the asyncio loop via `loop.call_soon_threadsafe` for upstream sends when state is LISTENING. +- A playback worker (asyncio task) drains an `asyncio.Queue` and writes to the OutputStream in a thread executor. + +### Reconnect & heartbeat + +- Pi → backend WS reconnects with exponential backoff 1 → 30 s. +- While disconnected, state stays IDLE; wakewords are ignored. +- An in-flight conversation does **not** survive a network drop — backend marks conversation `EndReason="error"`. +- Client sends `{"type":"ping"}` every 15 s when IDLE; backend uses it to update `Devices.LastSeenAt` and drive the UI online indicator. + +## Wire protocol (Pi ↔ backend) + +One persistent WebSocket at `wss:///device`. Authenticated via `Authorization: Bearer ` on the upgrade request; connection rejected if the token is revoked or unknown. + +### Binary frames + +Raw PCM16 LE, 24 kHz mono, exactly 1920 samples (3840 bytes). No header. Direction inferred from sender. Backend drops binary frames received outside an active session. Wrong-sized frames are dropped silently with a log entry (protects against runaways without surfacing an error). + +### Text frames (JSON envelopes) + +Client → backend: + +| `type` | Fields | Notes | +| ----------------- | ----------------------------------- | ----------------------------------------------------------- | +| `hello` | `client_version`, `device_id` | Sent on connect; backend replies `hello_ack` with the device's config snapshot. | +| `ping` | — | Every 15 s while IDLE. Backend replies `pong`. | +| `wake` | `at` (client ms timestamp) | Triggers backend to open upstream Realtime WS. | +| `cancel` | — | User-initiated abort (e.g. SIGINT). Backend closes session. | +| `playback_done` | `for` (`wake` \| `sleep` \| `assistant_turn`) | Lets backend know when audio finished playing on the Pi (needed to time `session_ended` after `sleep.wav`). | +| `tool_result` | `call_id`, `ok`, `result?`, `error?` | Reply to a Pi-side tool call (e.g. `set_volume`). | + +Backend → client: + +| `type` | Fields | Notes | +| ------------------ | ----------------------------------- | ---------------------------------------------------------- | +| `hello_ack` | `config` (voice/model/prompt/tools/idle_timeout) | Client uses this to set local state. | +| `pong` | — | | +| `config_updated` | `config` | Pushed when admin saves changes in the UI; takes effect next session. | +| `session_started` | `conversation_id` | Realtime upstream is ready; client may start uplink. | +| `assistant_done` | — | Assistant turn ended; client unmutes uplink. | +| `session_ended` | `reason` (`tool` \| `idle` \| `error`) | Client plays `sleep.wav` (if present), then `playback_done(for=sleep)`. | +| `tool_call` | `call_id`, `name`, `arguments` | Invoked for tools whose execution requires the Pi. | +| `play_audio` | `kind` (`session` \| `notification`) | Marker; subsequent binary frames are payload. Ended by `play_audio_end`. | +| `play_audio_end` | — | End-of-payload marker for `play_audio(kind=notification)`. | +| `error` | `code`, `message`, `fatal?` | Logged on client; non-fatal unless `fatal=true`. | + +Uplink muting is purely a client-side decision based on state. Backend doesn't enforce it. + +## Conversation lifecycle + +``` +PI BACKEND OPENAI REALTIME +───────────────────────────────────────────────────────────────────────────────── +state=IDLE +mic frames → wakeword.predict +wakeword fires +play wake.wav (non-blocking) +send {type:"wake"} ──► + open WSS to OpenAI ──► + send session.update ──► + (voice, instructions, + input_audio_transcription, + tools=[end_session, + get_current_time, + set_volume]) + ◄── session.updated + insert Conversations row + send {type:"session_started", + conversation_id} ──► +state=LISTENING +mic frames (binary) ──────────────► forward as input_audio_buffer.append ─► + ◄── input_audio_buffer.speech_started + ◄── input_audio_buffer.speech_stopped + ◄── response.created + ◄── response.audio.delta (x N) + binary frames ◄─────────────── forward to PI as binary +binary frames ──► playback queue +state=ASSISTANT_SPEAKING +(uplink muted) + ◄── response.audio_transcript.done + (persist assistant Turn) + ◄── response.done + send {type:"assistant_done"} ──► +state=LISTENING + ◄── conversation.item.input_audio_transcription.completed + (persist user Turn) +...follow-up turn(s)... + + Path A — model calls end_session + ◄── response.function_call_arguments.done + (name="end_session") + ◄── response.audio.delta (the "bye" reply) + forward audio to PI ──────────► + wait for response.done + ◄── response.done + send {type:"assistant_done"} ──► + send {type:"session_ended", + reason="tool"} ──► + close upstream WSS + persist Conversations.EndReason +state=IDLE-PENDING +wait for current audio buffer drain +play sleep.wav +send {type:"playback_done", + for:"sleep"} ──► +state=IDLE +(wakeword re-enabled) + + Path B — idle timeout +no mic activity for IdleTimeoutSeconds while LISTENING + send {type:"session_ended", + reason="idle"} ──► + close upstream WSS +(same sleep.wav + transition as Path A) +``` + +### `end_session` sequencing + +The tool call arrives *during* the assistant's response. The relay must: + +1. Forward all audio deltas until `response.done`. +2. Then send `assistant_done`, then `session_ended(reason=tool)`. +3. NOT execute the tool mid-response — `end_session` has no real side effect; the relay records "the model asked to end" and closes after the audio finishes. The tool result returned to OpenAI is `{"ok": true}` (for cleanliness; the upstream is about to close). + +## Tools registry + +C# interface, registered as singletons at startup: + +```csharp +public interface ITool +{ + string Name { get; } // OpenAI tool name + string Description { get; } // shown to the model + JsonSchema ParameterSchema { get; } // OpenAI-compatible + Task ExecuteAsync( + ToolInvocation invocation, + DeviceContext device, + CancellationToken ct); +} + +public record ToolInvocation(string CallId, JsonElement Arguments); +public record ToolResult(JsonElement Output); // Output is JSON sent back to model +public record DeviceContext(Guid DeviceId, Guid ConversationId, IDeviceChannel Channel); +``` + +`IDeviceChannel` lets a tool send `tool_call`-style messages to the Pi and await `tool_result`. Tools that don't need the Pi simply don't use it. + +### v1 built-in tools + +1. **`end_session`** — no parameters. Sets a "model wants to end" flag on the active `RealtimeSession`; the relay's `response.done` handler closes the upstream WS and emits `session_ended(reason=tool)`. Returns `{"ok": true}`. + +2. **`get_current_time`** — no parameters. Returns `{"now": ""}` from `DateTimeOffset.UtcNow`. Smoke test for the tool plane. + +3. **`set_volume`** — `level: integer 0..100`. Backend sends `tool_call(name=set_volume, arguments={level})` to the Pi; Pi runs `amixer set 'PCM' N%` (or device-specific mixer) and replies with `tool_result(call_id, ok, error?)`. Backend forwards back to OpenAI as the function call output. + +The per-device enabled set is enforced by filtering the tool list in `session.update`. Disabled tools are simply not advertised to the model. + +## Data model (SQLite via EF Core) + +``` +AspNetUsers, AspNetRoles, ... # Identity standard +Devices + Id (Guid) OwnerUserId Name PairedAt LastSeenAt TokenHash IsRevoked +DeviceConfigs # 1:1 with Devices + DeviceId SystemPrompt Voice Model IdleTimeoutSeconds EnabledToolsJson +PairingCodes + Code (8 chars) OwnerUserId ExpiresAt ConsumedAt +Conversations + Id DeviceId StartedAt EndedAt EndReason ("tool" | "idle" | "error") +Turns + Id ConversationId Index Role ("user" | "assistant" | "tool") Text ToolName ToolArgsJson ToolResultJson CreatedAt +SystemSettings # singleton row + DefaultVoice DefaultModel DefaultSystemPrompt +``` + +- `TokenHash` = SHA-256 of the device token; plaintext is shown once during pairing and never persisted. +- `EnabledToolsJson` is a small JSON array of tool names; no relational table needed. +- `Turns.Index` is monotonic per-conversation for ordering. +- `Database.Migrate()` runs on startup; SQLite file at `/data/assistant.db` on a Coolify named volume. + +## Pairing flow + +UI side: + +1. User signs in, clicks **Add device**, enters a friendly name. +2. Backend generates an 8-char alphanumeric code from a 32-char alphabet (no `0/O/1/l/I`); stores `PairingCodes(Code, OwnerUserId, ExpiresAt = NOW + 10 min)`. +3. UI shows the code and the one-line installer: + ``` + curl -fsSL https:///install.sh | bash + ``` +4. Code stays visible on the device page until consumed or expired; regenerate at any time. + +Pi side, first run: + +1. Installer runs. +2. If `~/assistant/state/config.json` exists with a valid `device_token`, **skip pairing** and proceed to update path. +3. Otherwise prompt `Pairing code: `. +4. `POST https:///api/pair` with `{ "code", "hostname", "client_version" }`. +5. Backend validates code (exists, unexpired, unconsumed), creates `Devices(OwnerUserId = code.OwnerUserId, Name = , TokenHash = sha256(newToken))`, marks code `ConsumedAt = NOW`, creates default `DeviceConfigs` row from `SystemSettings`, returns `{ "device_id", "device_token", "backend_ws" }`. +6. Installer writes `~/assistant/state/config.json` (mode 0600). + +Security: +- Pairing codes are one-shot, 10-min TTL, 8 chars from a 32-char alphabet (~40 bits — fine for the TTL window). +- Device tokens are 32 bytes random, base64url-encoded, stored only as SHA-256 hash server-side. + +## Install script + +`install.sh` is served by the backend at `GET /install.sh` and is fully idempotent. + +``` +1. Resolve BACKEND_URL from the script's own served-from origin (no hardcoding). +2. apt-get install -y python3-venv libportaudio2 alsa-utils + (sudo prompt OK; passwordless-sudo per findings.md §E is the recommended setup). +3. mkdir -p ~/assistant/{code,state} +4. python3 -m venv ~/assistant/.venv (if missing) +5. curl -fsSL $BACKEND_URL/client.tar.gz | tar -xz -C ~/assistant/code +6. ~/assistant/.venv/bin/pip install -r ~/assistant/code/requirements.txt +7. ~/assistant/.venv/bin/pip install --no-deps -r ~/assistant/code/requirements-nodeps.txt + (openwakeword install split per findings.md §3). +8. If config.json missing → run `~/assistant/.venv/bin/python -m client.pair`, + which prompts for the code and writes config.json. +9. Install systemd user unit at ~/.config/systemd/user/assistant.service. +10. systemctl --user daemon-reload +11. systemctl --user enable --now assistant +12. loginctl enable-linger pi (so the unit runs without an active login). +13. Print status; if pairing happened, print "Paired to as device ". +``` + +The systemd user unit runs `~/assistant/.venv/bin/python -m client.main` with: +- `Restart=on-failure` +- `RestartSec=5` +- `Environment=PA_ALSA_PLUGHW=1` + +### Updates + +Re-running the same one-liner upgrades the client. Steps 5–7 fetch and reinstall the latest bundle; step 11 restarts the unit. Pairing is skipped because `config.json` already exists. + +`curl ... | bash -s -- --reset` deletes `config.json` and forces a fresh pairing. + +### Backend install endpoints + +- `GET /install.sh` — returns the shell script with `BACKEND_URL` baked from the request's `X-Forwarded-Host` / `Host`. Public, no auth. +- `GET /client.tar.gz` — returns the current client bundle (built at deploy time into `wwwroot/`). Public, no auth — the code is not a secret; the device token gates the device WS. +- `POST /api/pair` — accepts pairing code, returns device token. No prior auth (code is the credential). + +The bundle is built during the Docker image build: a stage runs `tar czf client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt` and copies the result into `wwwroot/`. + +## Error handling + +| Scenario | Detection | Backend action | Client action | +| --- | --- | --- | --- | +| Pi loses network | WS close / heartbeat miss | Close upstream Realtime WS if active, log conversation `EndReason="error"`, mark device offline | Exponential backoff reconnect 1→30 s, stay IDLE, no wakeword | +| Backend restarts | Pi WS closes from server side | (n/a — process gone) | Same backoff reconnect; on reconnect send fresh `hello` | +| OpenAI WS drops mid-conversation | upstream `on_close` | Send `error(code=upstream)` + `session_ended(reason=error)`, persist partial transcript | Play `sleep.wav`, return to IDLE | +| OpenAI rate limit / 429 on connect | upstream open fails | Don't open session; send `error(code=upstream, message=...)` instead of `session_started` | Play `sleep.wav`, IDLE | +| Tool execution exception (server-side) | C# try/catch around `ITool.ExecuteAsync` | Send tool result `{"error": "..."}` back to OpenAI so the model can recover; log | (no client involvement) | +| Pi-side tool fails (e.g. `set_volume` amixer error) | `tool_result.ok=false` | Forward `{"error": "..."}` to OpenAI | Log and continue | +| Device token revoked while connected | Admin clicks revoke | Set `IsRevoked=true`, close all WS for that device with `error(code=auth_revoked, fatal=true)` | Stop the process (systemd will restart, fail to auth, sleep 60 s, retry — visible in logs) | +| SQLite write failure | EF exception | Bubble up to REST as 500; on the hub side, drop the offending log event, keep session running | (n/a) | +| Two simultaneous wakeword fires | `wake` while not IDLE | Backend ignores second `wake` (idempotent within active session) | Client also gates locally | +| Audio buffer overflow on Pi | PortAudio callback queue full | (n/a) | Drop oldest frame, log a warning | + +## Management UI (v1) + +React SPA, served from the backend's `wwwroot/`. Pages: + +- **Sign in / sign up** (ASP.NET Identity). +- **Devices** — list of the signed-in user's devices: name, online indicator, last-seen. +- **Device detail** — rename; generate pairing code (with countdown); revoke; per-device config form: system prompt override, voice, model, idle timeout slider, enabled-tools checkboxes; conversation history viewer (paginated, text only). +- **Admin** (admin role only) — user list, invite/disable users, global model/voice/prompt defaults. + +Out of v1: live conversation tail, audio meters, audio recordings, memory UI, per-device `wake.wav`/`sleep.wav` upload (filesystem only for v1; v1.1 adds upload). + +## Testing strategy + +- **Backend unit tests** (xUnit): `ToolRegistry` filtering by enabled set; `PairingService` (code generation, expiry, single-use); `ConversationLog` (turn ordering, persistence); auth/role enforcement on the admin endpoints. +- **Backend integration tests** (`WebApplicationFactory`): full pair flow end-to-end against in-memory SQLite; device-hub WS with a fake OpenAI upstream (test double that emits scripted Realtime events). +- **Client unit tests** (pytest): state machine transitions (pure-Python, no audio); resample correctness against golden samples; config loading. Audio I/O and wakeword are not unit-tested — already covered by the prototypes and by manual end-to-end. +- **End-to-end manual smoke test** (in repo `README.md`): pair a fresh Pi, say wakeword, ask "what time is it", say "bye", verify conversation row + sleep.wav heard. Run after any change that touches the conversation flow. +- No load / perf testing. + +## Deployment (Coolify) + +Single Docker image, multi-stage Dockerfile (`preparing-dotnet-react-app-for-coolify` skill is the reference for the layout): + +1. **Frontend build** stage — Node, `npm ci && npm run build` against `frontend/`. +2. **Backend build** stage — .NET SDK, `dotnet publish backend/ -c Release -o /publish`. +3. **Client bundle** stage — `tar czf /publish/wwwroot/client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt`. +4. **Final** stage — `mcr.microsoft.com/dotnet/aspnet:9.0`, copies `/publish`, copies SPA `dist/` into `wwwroot/`, sets `ASPNETCORE_URLS=http://+:8080`, exposes 8080. + +Coolify app (provisioned via `deploying-to-coolify-via-api`): + +- Source: this repo on Gitea. +- Expose 8080 (Traefik fronts it). +- `SERVICE_FQDN_BACKEND` → backend domain. +- Env vars: + - `OPENAI_API_KEY` — secret. + - `ConnectionStrings__Default=Data Source=/data/assistant.db` + - `ASPNETCORE_URLS=http://+:8080` +- Named volumes: + - `/data` — SQLite file. + - `/keys` — ASP.NET data-protection keys (cookies survive container restarts). +- Deploys triggered fire-and-forget via the Coolify v4 API (per global CLAUDE.md rule). + +## Repo layout (final) + +``` +Assistant/ +├── CLAUDE.md +├── README.md # smoke-test instructions +├── findings.md # prototype-phase findings (kept) +├── tests/ # prototype throwaways (kept for reference) +│ ├── 01-record-play/ +│ └── 02-wakeword/ +├── docs/superpowers/ +│ ├── specs/ +│ │ ├── 2026-06-11-voice-assistant-prototypes-design.md +│ │ └── 2026-06-11-smart-assistant-design.md # this design +│ └── plans/ +│ └── (one or more implementation plans, written next) +├── backend/ # ASP.NET solution +│ ├── backend.csproj +│ ├── Program.cs +│ ├── Auth/ Devices/ DeviceHub/ Realtime/ Tools/ Conversations/ Install/ Data/ +│ └── wwwroot/ # filled at build: SPA dist + client.tar.gz +├── frontend/ # React + Vite source +│ ├── package.json +│ └── src/ +├── client/ # Python package +│ ├── requirements.txt # sounddevice, numpy, scipy, onnxruntime, websockets, ... +│ ├── requirements-nodeps.txt # openwakeword +│ ├── main.py audio.py wakeword.py session.py playback.py state.py config.py pair.py log.py +│ └── systemd/assistant.service +├── Dockerfile +└── deploy.json # Coolify config (project/server uuids, domain) +``` + +## Open items deferred to implementation plans + +- Exact OpenAI Realtime model + voice default (`SystemSettings.DefaultModel`/`DefaultVoice`) — pick at implementation time from currently-available choices. +- Whether wake.wav / sleep.wav playback uses the persistent OutputStream (preferred — same path as session audio) or a separate one-shot — settled in the client plan. +- ASP.NET data-protection key seeding (initial bootstrap on first run). +- React UI component library — Tailwind only, or Tailwind + a small headless component lib (Radix / shadcn). +- Exact mixer control name for `set_volume` on the LISTENAI device — confirm with `amixer scontrols` on the Pi during implementation. + +## Reference paths + +- Prototype spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md` +- Prototype findings: `findings.md` +- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`) +- Coolify skills: `deploying-to-coolify-via-api`, `preparing-dotnet-react-app-for-coolify`