# Smart Assistant — design **Date:** 2026-06-11 **Status:** approved (brainstorming complete) **Scope:** main build of the voice assistant, replacing the throwaway prototypes in `tests/`. Two deliverables: a Python client on the Raspberry Pi, and an ASP.NET backend (with embedded React management UI) deployed to Coolify. Brings up everything needed to wake on a wakeword, hold a multi-turn conversation through the OpenAI Realtime API, persist a text history per device, run a small built-in tool registry, and let users manage their devices through a browser. ## Context The prototype phase (`docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`, results in `findings.md`) proved that recording/playback and openwakeword both work on the target Pi. The main assistant is built on top of those findings. Key carry-forwards from the prototypes: - USB Speaker Phone (ALSA card 3, Anhui LISTENAI) is the only mic/speaker. 48 kHz native; `PA_ALSA_PLUGHW=1` set before importing `sounddevice` lets us open at any rate. - `python-sounddevice` + `numpy` are the audio stack. - `openwakeword` runs at ~0.6 s load, well under 50% of one core; the `alexa` stock model is reused for v1 (custom-trained wakeword is a separate follow-up task). - `openwakeword==0.6.0` needs `--no-deps` install on Python 3.13 because its `tflite-runtime` pin has no wheel; the prototype's two-pass install pattern is kept. - Speaker hears its own output → detector must be gated while playing audio, and uplink to OpenAI must be muted while the assistant is speaking. - TTS/beep must not block the input stream — callback-driven InputStream + persistent OutputStream is the pattern. ## Goals 1. A fresh Raspberry Pi can be paired to a user account by running one curl-piped install command and entering a short code. 2. The Pi wakes on the wakeword, opens a Realtime conversation, supports multi-turn until the user says "bye" or goes idle, and plays optional `wake.wav` / `sleep.wav` cues. 3. A signed-in user sees only their own devices and conversations in the management UI. 4. An admin user can manage system-wide defaults and other users. 5. Per-device configuration: name, system prompt override, voice, model, idle timeout, enabled-tools toggles. Changes take effect on the next session. 6. Text-only conversation history per device, viewable in the UI. 7. Built-in C# tool registry with `end_session`, `get_current_time`, `set_volume` shipped in v1. 8. Backend deployed to Coolify as one container with one domain; deploys triggered fire-and-forget via the Coolify v4 API. 9. The install script is the update mechanism: re-running it upgrades the client in place without re-pairing. ## Non-goals (v1) - Long-term memory across sessions. - MCP-based external tools (only built-in C# tools). - Audio recordings of conversations (text only). - Live conversation tailing or audio meters in the UI. - Per-device upload of `wake.wav`/`sleep.wav` via UI (filesystem only — v1.1). - Custom-trained wakeword (uses the stock `alexa` model for v1). - Barge-in (interrupting the assistant mid-reply). - Multi-language (English only). - Offline / local-only fallback when the network is down. - OIDC sign-in (local ASP.NET Identity only). - `.deb` packaging for the client. - Load / performance testing. ## Architecture ### Topology ``` ┌─────────────────────┐ pair code / token ┌──────────────────────────────────────────┐ │ Browser (admin UI) │ ──────HTTPS (REST)────────▶ │ │ │ React SPA │ │ ASP.NET backend (Coolify container) │ └─────────────────────┘ │ │ │ ┌────────────┐ ┌────────────────────┐ │ ┌─────────────────────┐ WSS audio + control │ │ REST/UI │ │ Device hub │ │ │ Pi client (Python) │ ◀───────────────────────────▶│ │ + Identity│ │ (per-device WS) │ │ │ wakeword + audio │ │ └────────────┘ └─────────┬──────────┘ │ └─────────────────────┘ │ │ │ │ ┌────────────┐ ┌────────▼──────────┐ │ │ │ Tools │ │ Realtime relay │ │ │ │ registry │◀─┤ (per-session WS │ │ │ └────────────┘ │ upstream) │ │ │ └─────────┬──────────┘ │ │ SQLite (named volume) │ │ └────────────────────────────┼────────────┘ │ WSS ▼ OpenAI Realtime API ``` One container; React SPA served by the same ASP.NET process. SQLite file lives on a Coolify named volume. The OpenAI API key is held server-side; clients never see it. ### Why backend relay (not direct Pi ↔ OpenAI) - One API key, server-side only. - All user-side and assistant-side transcripts land in our database directly off the upstream WS events (no separate STT pass). - Tool definitions and changes don't require a client redeploy. - Backend can push audio to a Pi on its own (future feature) over the same WS. ### Backend Single ASP.NET 9 project (`backend/`) organised into modules: ``` backend/ ├── Program.cs # composition root, EF migrate, route + hub mapping ├── Auth/ # ASP.NET Identity, roles (Admin, User), invite flow ├── Devices/ # device CRUD, pairing-code issuance, token mgmt ├── DeviceHub/ # WebSocket handler at /device, per-device session state ├── Realtime/ # RealtimeRelay: per-conversation upstream WS to OpenAI ├── Tools/ # ITool interface, registry, built-in tools ├── Conversations/ # persistence of turns, transcripts, tool calls ├── Install/ # serves install.sh and client.tar.gz ├── Data/ # EF Core DbContext, migrations, SQLite └── wwwroot/ # built React SPA (Vite output) + client.tar.gz ``` Key services (DI): - `DeviceRegistry` (singleton) — in-memory map `deviceId → ActiveDevice`, tracks online state and last frame timestamp. - `RealtimeSessionFactory` (singleton) — given a `DeviceContext`, opens an upstream WSS to OpenAI and returns a `RealtimeSession` instance bound to one conversation. - `ToolRegistry` (singleton) — static list of `ITool`s registered at startup, filterable by per-device enabled set. - `ConversationLog` (scoped) — append-only writer for turns; flushes per event so a crash mid-conversation still preserves what was said. - `OpenAIKeyProvider` (singleton) — reads `OPENAI_API_KEY` from env. ### Pi client Python package (`client/`), deployed to `~/assistant/code/` on the Pi: ``` client/ ├── main.py # entry: sets PA_ALSA_PLUGHW BEFORE importing sounddevice ├── audio.py # USB device discovery, persistent in/out streams ├── wakeword.py # openwakeword wrapper + gating flag ├── session.py # backend WS client: binary audio + JSON control ├── playback.py # queue-fed playback, wake/sleep wav hooks ├── state.py # state machine ├── config.py # loads ~/assistant/state/config.json ├── pair.py # one-shot: prompt for code, hit /api/pair, write config.json └── log.py # structured logging to journald ``` State persists at `~/assistant/state/`: - `config.json` (mode 0600) — `device_id`, `device_token`, `backend_ws` - `wake.wav` / `sleep.wav` — optional cues, played if present - (room for future per-device assets; never rsynced/overwritten by deploys) `~/assistant/code/` is the only directory that the install script replaces on update; `~/assistant/state/` is untouched. ### State machine ``` wakeword fires session_started ┌─────────────────────┐ ┌──────────────────────┐ │ ▼ │ ▼ [IDLE] ─────────► [WAKE_PENDING] ─────► [LISTENING] ───────► [ASSISTANT_SPEAKING] ▲ │ ▲ │ │ │ │ assistant done │ │ └───┴──────────────────┘ │ session_ended └─────────────────────────────── (idle timeout OR end_session tool) plays sleep.wav, re-enables wakeword ``` - **IDLE** — wakeword detector enabled, no upstream audio. Persistent backend WS open with heartbeats. - **WAKE_PENDING** — wakeword fired; play `wake.wav` (non-blocking); send `wake`; wait for `session_started`. Wakeword disabled. - **LISTENING** — mic frames stream to backend; server-VAD on OpenAI decides turn boundaries; we don't compute local VAD. - **ASSISTANT_SPEAKING** — uplink muted; inbound frames flow to playback queue. - On `assistant_done` from backend → LISTENING again (uplink unmuted). - On `session_ended` from backend → wait for current audio to drain, play `sleep.wav`, send `playback_done(for=sleep)` → IDLE. ### Audio plumbing - **Wire format**: PCM16 little-endian, 24 kHz mono, exactly 1920 samples (3840 bytes, 80 ms) per binary frame. Matches OpenAI Realtime's default `pcm16`. No transcoding in the backend. - **Capture**: single InputStream, callback-driven, blocksize 1920, sample rate 24 000 Hz. Callback only enqueues — never blocks. - **Playback**: single OutputStream opened at startup, persistent. Playback worker pulls from an `asyncio.Queue` and writes — never `sd.play()/sd.wait()` (which would block the InputStream the way the prototype did). - **Wakeword feed**: each captured 1920-sample frame is downsampled to 1280 samples (80 ms at 16 kHz) via `scipy.signal.resample_poly(frame, up=2, down=3)` and fed to `model.predict()`. - `PA_ALSA_PLUGHW=1` is set at the very top of `main.py` before any `sounddevice` import. ### Threading model - Main thread runs an asyncio loop owning the backend WS. - PortAudio invokes the InputStream callback on its own thread; the callback writes mic frames to a bounded `queue.Queue`. On overflow, oldest frame is dropped and a warning is logged. - A bridge thread reads `queue.Queue`, runs wakeword.predict() when state is IDLE/WAKE_PENDING, and forwards bytes to the asyncio loop via `loop.call_soon_threadsafe` for upstream sends when state is LISTENING. - A playback worker (asyncio task) drains an `asyncio.Queue` and writes to the OutputStream in a thread executor. ### Reconnect & heartbeat - Pi → backend WS reconnects with exponential backoff 1 → 30 s. - While disconnected, state stays IDLE; wakewords are ignored. - An in-flight conversation does **not** survive a network drop — backend marks conversation `EndReason="error"`. - Client sends `{"type":"ping"}` every 15 s when IDLE; backend uses it to update `Devices.LastSeenAt` and drive the UI online indicator. ## Wire protocol (Pi ↔ backend) One persistent WebSocket at `wss:///device`. Authenticated via `Authorization: Bearer ` on the upgrade request; connection rejected if the token is revoked or unknown. ### Binary frames Raw PCM16 LE, 24 kHz mono, exactly 1920 samples (3840 bytes). No header. Direction inferred from sender. Backend drops binary frames received outside an active session. Wrong-sized frames are dropped silently with a log entry (protects against runaways without surfacing an error). ### Text frames (JSON envelopes) Client → backend: | `type` | Fields | Notes | | ----------------- | ----------------------------------- | ----------------------------------------------------------- | | `hello` | `client_version`, `device_id` | Sent on connect; backend replies `hello_ack` with the device's config snapshot. | | `ping` | — | Every 15 s while IDLE. Backend replies `pong`. | | `wake` | `at` (client ms timestamp) | Triggers backend to open upstream Realtime WS. | | `cancel` | — | User-initiated abort (e.g. SIGINT). Backend closes session. | | `playback_done` | `for` (`wake` \| `sleep` \| `assistant_turn`) | Lets backend know when audio finished playing on the Pi (needed to time `session_ended` after `sleep.wav`). | | `tool_result` | `call_id`, `ok`, `result?`, `error?` | Reply to a Pi-side tool call (e.g. `set_volume`). | Backend → client: | `type` | Fields | Notes | | ------------------ | ----------------------------------- | ---------------------------------------------------------- | | `hello_ack` | `config` (voice/model/prompt/tools/idle_timeout) | Client uses this to set local state. | | `pong` | — | | | `config_updated` | `config` | Pushed when admin saves changes in the UI; takes effect next session. | | `session_started` | `conversation_id` | Realtime upstream is ready; client may start uplink. | | `assistant_done` | — | Assistant turn ended; client unmutes uplink. | | `session_ended` | `reason` (`tool` \| `idle` \| `error`) | Client plays `sleep.wav` (if present), then `playback_done(for=sleep)`. | | `tool_call` | `call_id`, `name`, `arguments` | Invoked for tools whose execution requires the Pi. | | `play_audio` | `kind` (`session` \| `notification`) | Marker; subsequent binary frames are payload. Ended by `play_audio_end`. | | `play_audio_end` | — | End-of-payload marker for `play_audio(kind=notification)`. | | `error` | `code`, `message`, `fatal?` | Logged on client; non-fatal unless `fatal=true`. | Uplink muting is purely a client-side decision based on state. Backend doesn't enforce it. ## Conversation lifecycle ``` PI BACKEND OPENAI REALTIME ───────────────────────────────────────────────────────────────────────────────── state=IDLE mic frames → wakeword.predict wakeword fires play wake.wav (non-blocking) send {type:"wake"} ──► open WSS to OpenAI ──► send session.update ──► (voice, instructions, input_audio_transcription, tools=[end_session, get_current_time, set_volume]) ◄── session.updated insert Conversations row send {type:"session_started", conversation_id} ──► state=LISTENING mic frames (binary) ──────────────► forward as input_audio_buffer.append ─► ◄── input_audio_buffer.speech_started ◄── input_audio_buffer.speech_stopped ◄── response.created ◄── response.audio.delta (x N) binary frames ◄─────────────── forward to PI as binary binary frames ──► playback queue state=ASSISTANT_SPEAKING (uplink muted) ◄── response.audio_transcript.done (persist assistant Turn) ◄── response.done send {type:"assistant_done"} ──► state=LISTENING ◄── conversation.item.input_audio_transcription.completed (persist user Turn) ...follow-up turn(s)... Path A — model calls end_session ◄── response.function_call_arguments.done (name="end_session") ◄── response.audio.delta (the "bye" reply) forward audio to PI ──────────► wait for response.done ◄── response.done send {type:"assistant_done"} ──► send {type:"session_ended", reason="tool"} ──► close upstream WSS persist Conversations.EndReason state=IDLE-PENDING wait for current audio buffer drain play sleep.wav send {type:"playback_done", for:"sleep"} ──► state=IDLE (wakeword re-enabled) Path B — idle timeout no mic activity for IdleTimeoutSeconds while LISTENING send {type:"session_ended", reason="idle"} ──► close upstream WSS (same sleep.wav + transition as Path A) ``` ### `end_session` sequencing The tool call arrives *during* the assistant's response. The relay must: 1. Forward all audio deltas until `response.done`. 2. Then send `assistant_done`, then `session_ended(reason=tool)`. 3. NOT execute the tool mid-response — `end_session` has no real side effect; the relay records "the model asked to end" and closes after the audio finishes. The tool result returned to OpenAI is `{"ok": true}` (for cleanliness; the upstream is about to close). ## Tools registry C# interface, registered as singletons at startup: ```csharp public interface ITool { string Name { get; } // OpenAI tool name string Description { get; } // shown to the model JsonSchema ParameterSchema { get; } // OpenAI-compatible Task ExecuteAsync( ToolInvocation invocation, DeviceContext device, CancellationToken ct); } public record ToolInvocation(string CallId, JsonElement Arguments); public record ToolResult(JsonElement Output); // Output is JSON sent back to model public record DeviceContext(Guid DeviceId, Guid ConversationId, IDeviceChannel Channel); ``` `IDeviceChannel` lets a tool send `tool_call`-style messages to the Pi and await `tool_result`. Tools that don't need the Pi simply don't use it. ### v1 built-in tools 1. **`end_session`** — no parameters. Sets a "model wants to end" flag on the active `RealtimeSession`; the relay's `response.done` handler closes the upstream WS and emits `session_ended(reason=tool)`. Returns `{"ok": true}`. 2. **`get_current_time`** — no parameters. Returns `{"now": ""}` from `DateTimeOffset.UtcNow`. Smoke test for the tool plane. 3. **`set_volume`** — `level: integer 0..100`. Backend sends `tool_call(name=set_volume, arguments={level})` to the Pi; Pi runs `amixer set 'PCM' N%` (or device-specific mixer) and replies with `tool_result(call_id, ok, error?)`. Backend forwards back to OpenAI as the function call output. The per-device enabled set is enforced by filtering the tool list in `session.update`. Disabled tools are simply not advertised to the model. ## Data model (SQLite via EF Core) ``` AspNetUsers, AspNetRoles, ... # Identity standard Devices Id (Guid) OwnerUserId Name PairedAt LastSeenAt TokenHash IsRevoked DeviceConfigs # 1:1 with Devices DeviceId SystemPrompt Voice Model IdleTimeoutSeconds EnabledToolsJson PairingCodes Code (8 chars) OwnerUserId ExpiresAt ConsumedAt Conversations Id DeviceId StartedAt EndedAt EndReason ("tool" | "idle" | "error") Turns Id ConversationId Index Role ("user" | "assistant" | "tool") Text ToolName ToolArgsJson ToolResultJson CreatedAt SystemSettings # singleton row DefaultVoice DefaultModel DefaultSystemPrompt ``` - `TokenHash` = SHA-256 of the device token; plaintext is shown once during pairing and never persisted. - `EnabledToolsJson` is a small JSON array of tool names; no relational table needed. - `Turns.Index` is monotonic per-conversation for ordering. - `Database.Migrate()` runs on startup; SQLite file at `/data/assistant.db` on a Coolify named volume. ## Pairing flow UI side: 1. User signs in, clicks **Add device**, enters a friendly name. 2. Backend generates an 8-char alphanumeric code from a 32-char alphabet (no `0/O/1/l/I`); stores `PairingCodes(Code, OwnerUserId, ExpiresAt = NOW + 10 min)`. 3. UI shows the code and the one-line installer: ``` curl -fsSL https:///install.sh | bash ``` 4. Code stays visible on the device page until consumed or expired; regenerate at any time. Pi side, first run: 1. Installer runs. 2. If `~/assistant/state/config.json` exists with a valid `device_token`, **skip pairing** and proceed to update path. 3. Otherwise prompt `Pairing code: `. 4. `POST https:///api/pair` with `{ "code", "hostname", "client_version" }`. 5. Backend validates code (exists, unexpired, unconsumed), creates `Devices(OwnerUserId = code.OwnerUserId, Name = , TokenHash = sha256(newToken))`, marks code `ConsumedAt = NOW`, creates default `DeviceConfigs` row from `SystemSettings`, returns `{ "device_id", "device_token", "backend_ws" }`. 6. Installer writes `~/assistant/state/config.json` (mode 0600). Security: - Pairing codes are one-shot, 10-min TTL, 8 chars from a 32-char alphabet (~40 bits — fine for the TTL window). - Device tokens are 32 bytes random, base64url-encoded, stored only as SHA-256 hash server-side. ## Install script `install.sh` is served by the backend at `GET /install.sh` and is fully idempotent. ``` 1. Resolve BACKEND_URL from the script's own served-from origin (no hardcoding). 2. apt-get install -y python3-venv libportaudio2 alsa-utils (sudo prompt OK; passwordless-sudo per findings.md §E is the recommended setup). 3. mkdir -p ~/assistant/{code,state} 4. python3 -m venv ~/assistant/.venv (if missing) 5. curl -fsSL $BACKEND_URL/client.tar.gz | tar -xz -C ~/assistant/code 6. ~/assistant/.venv/bin/pip install -r ~/assistant/code/requirements.txt 7. ~/assistant/.venv/bin/pip install --no-deps -r ~/assistant/code/requirements-nodeps.txt (openwakeword install split per findings.md §3). 8. If config.json missing → run `~/assistant/.venv/bin/python -m client.pair`, which prompts for the code and writes config.json. 9. Install systemd user unit at ~/.config/systemd/user/assistant.service. 10. systemctl --user daemon-reload 11. systemctl --user enable --now assistant 12. loginctl enable-linger pi (so the unit runs without an active login). 13. Print status; if pairing happened, print "Paired to as device ". ``` The systemd user unit runs `~/assistant/.venv/bin/python -m client.main` with: - `Restart=on-failure` - `RestartSec=5` - `Environment=PA_ALSA_PLUGHW=1` ### Updates Re-running the same one-liner upgrades the client. Steps 5–7 fetch and reinstall the latest bundle; step 11 restarts the unit. Pairing is skipped because `config.json` already exists. `curl ... | bash -s -- --reset` deletes `config.json` and forces a fresh pairing. ### Backend install endpoints - `GET /install.sh` — returns the shell script with `BACKEND_URL` baked from the request's `X-Forwarded-Host` / `Host`. Public, no auth. - `GET /client.tar.gz` — returns the current client bundle (built at deploy time into `wwwroot/`). Public, no auth — the code is not a secret; the device token gates the device WS. - `POST /api/pair` — accepts pairing code, returns device token. No prior auth (code is the credential). The bundle is built during the Docker image build: a stage runs `tar czf client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt` and copies the result into `wwwroot/`. ## Error handling | Scenario | Detection | Backend action | Client action | | --- | --- | --- | --- | | Pi loses network | WS close / heartbeat miss | Close upstream Realtime WS if active, log conversation `EndReason="error"`, mark device offline | Exponential backoff reconnect 1→30 s, stay IDLE, no wakeword | | Backend restarts | Pi WS closes from server side | (n/a — process gone) | Same backoff reconnect; on reconnect send fresh `hello` | | OpenAI WS drops mid-conversation | upstream `on_close` | Send `error(code=upstream)` + `session_ended(reason=error)`, persist partial transcript | Play `sleep.wav`, return to IDLE | | OpenAI rate limit / 429 on connect | upstream open fails | Don't open session; send `error(code=upstream, message=...)` instead of `session_started` | Play `sleep.wav`, IDLE | | Tool execution exception (server-side) | C# try/catch around `ITool.ExecuteAsync` | Send tool result `{"error": "..."}` back to OpenAI so the model can recover; log | (no client involvement) | | Pi-side tool fails (e.g. `set_volume` amixer error) | `tool_result.ok=false` | Forward `{"error": "..."}` to OpenAI | Log and continue | | Device token revoked while connected | Admin clicks revoke | Set `IsRevoked=true`, close all WS for that device with `error(code=auth_revoked, fatal=true)` | Stop the process (systemd will restart, fail to auth, sleep 60 s, retry — visible in logs) | | SQLite write failure | EF exception | Bubble up to REST as 500; on the hub side, drop the offending log event, keep session running | (n/a) | | Two simultaneous wakeword fires | `wake` while not IDLE | Backend ignores second `wake` (idempotent within active session) | Client also gates locally | | Audio buffer overflow on Pi | PortAudio callback queue full | (n/a) | Drop oldest frame, log a warning | ## Management UI (v1) React SPA, served from the backend's `wwwroot/`. Pages: - **Sign in / sign up** (ASP.NET Identity). - **Devices** — list of the signed-in user's devices: name, online indicator, last-seen. - **Device detail** — rename; generate pairing code (with countdown); revoke; per-device config form: system prompt override, voice, model, idle timeout slider, enabled-tools checkboxes; conversation history viewer (paginated, text only). - **Admin** (admin role only) — user list, invite/disable users, global model/voice/prompt defaults. Out of v1: live conversation tail, audio meters, audio recordings, memory UI, per-device `wake.wav`/`sleep.wav` upload (filesystem only for v1; v1.1 adds upload). ## Testing strategy - **Backend unit tests** (xUnit): `ToolRegistry` filtering by enabled set; `PairingService` (code generation, expiry, single-use); `ConversationLog` (turn ordering, persistence); auth/role enforcement on the admin endpoints. - **Backend integration tests** (`WebApplicationFactory`): full pair flow end-to-end against in-memory SQLite; device-hub WS with a fake OpenAI upstream (test double that emits scripted Realtime events). - **Client unit tests** (pytest): state machine transitions (pure-Python, no audio); resample correctness against golden samples; config loading. Audio I/O and wakeword are not unit-tested — already covered by the prototypes and by manual end-to-end. - **End-to-end manual smoke test** (in repo `README.md`): pair a fresh Pi, say wakeword, ask "what time is it", say "bye", verify conversation row + sleep.wav heard. Run after any change that touches the conversation flow. - No load / perf testing. ## Deployment (Coolify) Single Docker image, multi-stage Dockerfile (`preparing-dotnet-react-app-for-coolify` skill is the reference for the layout): 1. **Frontend build** stage — Node, `npm ci && npm run build` against `frontend/`. 2. **Backend build** stage — .NET SDK, `dotnet publish backend/ -c Release -o /publish`. 3. **Client bundle** stage — `tar czf /publish/wwwroot/client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt`. 4. **Final** stage — `mcr.microsoft.com/dotnet/aspnet:9.0`, copies `/publish`, copies SPA `dist/` into `wwwroot/`, sets `ASPNETCORE_URLS=http://+:8080`, exposes 8080. Coolify app (provisioned via `deploying-to-coolify-via-api`): - Source: this repo on Gitea. - Expose 8080 (Traefik fronts it). - `SERVICE_FQDN_BACKEND` → backend domain. - Env vars: - `OPENAI_API_KEY` — secret. - `ConnectionStrings__Default=Data Source=/data/assistant.db` - `ASPNETCORE_URLS=http://+:8080` - Named volumes: - `/data` — SQLite file. - `/keys` — ASP.NET data-protection keys (cookies survive container restarts). - Deploys triggered fire-and-forget via the Coolify v4 API (per global CLAUDE.md rule). ## Repo layout (final) ``` Assistant/ ├── CLAUDE.md ├── README.md # smoke-test instructions ├── findings.md # prototype-phase findings (kept) ├── tests/ # prototype throwaways (kept for reference) │ ├── 01-record-play/ │ └── 02-wakeword/ ├── docs/superpowers/ │ ├── specs/ │ │ ├── 2026-06-11-voice-assistant-prototypes-design.md │ │ └── 2026-06-11-smart-assistant-design.md # this design │ └── plans/ │ └── (one or more implementation plans, written next) ├── backend/ # ASP.NET solution │ ├── backend.csproj │ ├── Program.cs │ ├── Auth/ Devices/ DeviceHub/ Realtime/ Tools/ Conversations/ Install/ Data/ │ └── wwwroot/ # filled at build: SPA dist + client.tar.gz ├── frontend/ # React + Vite source │ ├── package.json │ └── src/ ├── client/ # Python package │ ├── requirements.txt # sounddevice, numpy, scipy, onnxruntime, websockets, ... │ ├── requirements-nodeps.txt # openwakeword │ ├── main.py audio.py wakeword.py session.py playback.py state.py config.py pair.py log.py │ └── systemd/assistant.service ├── Dockerfile └── deploy.json # Coolify config (project/server uuids, domain) ``` ## Open items deferred to implementation plans - Exact OpenAI Realtime model + voice default (`SystemSettings.DefaultModel`/`DefaultVoice`) — pick at implementation time from currently-available choices. - Whether wake.wav / sleep.wav playback uses the persistent OutputStream (preferred — same path as session audio) or a separate one-shot — settled in the client plan. - ASP.NET data-protection key seeding (initial bootstrap on first run). - React UI component library — Tailwind only, or Tailwind + a small headless component lib (Radix / shadcn). - Exact mixer control name for `set_volume` on the LISTENAI device — confirm with `amixer scontrols` on the Pi during implementation. ## Reference paths - Prototype spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md` - Prototype findings: `findings.md` - Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`) - Coolify skills: `deploying-to-coolify-via-api`, `preparing-dotnet-react-app-for-coolify`