tes/Assistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Smart-assistant main-build design spec

Browse Source 
Captures the architecture for the post-prototype build: Pi Python client +
ASP.NET backend (with embedded React UI) on Coolify, SQLite, multi-turn
Realtime sessions with end_session/get_current_time/set_volume tools,
pairing-code device onboarding, and idempotent install/update flow.
This commit is contained in:
Assistant builder
2026-06-11 15:48:10 +00:00
parent 8bc08f451d
commit 99a9950cee
1 changed files with 516 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-06-11-smart-assistant-design.md
+516
View File
@@ -0,0 +1,516 @@
# Smart Assistant — design
**Date:** 2026-06-11
**Status:** approved (brainstorming complete)
**Scope:** main build of the voice assistant, replacing the throwaway prototypes in `tests/`. Two deliverables: a Python client on the Raspberry Pi, and an ASP.NET backend (with embedded React management UI) deployed to Coolify. Brings up everything needed to wake on a wakeword, hold a multi-turn conversation through the OpenAI Realtime API, persist a text history per device, run a small built-in tool registry, and let users manage their devices through a browser.
## Context
The prototype phase (`docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`, results in `findings.md`) proved that recording/playback and openwakeword both work on the target Pi. The main assistant is built on top of those findings.
Key carry-forwards from the prototypes:
- USB Speaker Phone (ALSA card 3, Anhui LISTENAI) is the only mic/speaker. 48 kHz native; `PA_ALSA_PLUGHW=1` set before importing `sounddevice` lets us open at any rate.
- `python-sounddevice` + `numpy` are the audio stack.
- `openwakeword` runs at ~0.6 s load, well under 50% of one core; the `alexa` stock model is reused for v1 (custom-trained wakeword is a separate follow-up task).
- `openwakeword==0.6.0` needs `--no-deps` install on Python 3.13 because its `tflite-runtime` pin has no wheel; the prototype's two-pass install pattern is kept.
- Speaker hears its own output → detector must be gated while playing audio, and uplink to OpenAI must be muted while the assistant is speaking.
- TTS/beep must not block the input stream — callback-driven InputStream + persistent OutputStream is the pattern.
## Goals
1. A fresh Raspberry Pi can be paired to a user account by running one curl-piped install command and entering a short code.
2. The Pi wakes on the wakeword, opens a Realtime conversation, supports multi-turn until the user says "bye" or goes idle, and plays optional `wake.wav` / `sleep.wav` cues.
3. A signed-in user sees only their own devices and conversations in the management UI.
4. An admin user can manage system-wide defaults and other users.
5. Per-device configuration: name, system prompt override, voice, model, idle timeout, enabled-tools toggles. Changes take effect on the next session.
6. Text-only conversation history per device, viewable in the UI.
7. Built-in C# tool registry with `end_session`, `get_current_time`, `set_volume` shipped in v1.
8. Backend deployed to Coolify as one container with one domain; deploys triggered fire-and-forget via the Coolify v4 API.
9. The install script is the update mechanism: re-running it upgrades the client in place without re-pairing.
## Non-goals (v1)
- Long-term memory across sessions.
- MCP-based external tools (only built-in C# tools).
- Audio recordings of conversations (text only).
- Live conversation tailing or audio meters in the UI.
- Per-device upload of `wake.wav`/`sleep.wav` via UI (filesystem only — v1.1).
- Custom-trained wakeword (uses the stock `alexa` model for v1).
- Barge-in (interrupting the assistant mid-reply).
- Multi-language (English only).
- Offline / local-only fallback when the network is down.
- OIDC sign-in (local ASP.NET Identity only).
- `.deb` packaging for the client.
- Load / performance testing.
## Architecture
### Topology
```
┌─────────────────────┐ pair code / token ┌──────────────────────────────────────────┐
│ Browser (admin UI) │ ──────HTTPS (REST)────────▶ │ │
│ React SPA │ │ ASP.NET backend (Coolify container) │
└─────────────────────┘ │ │
│ ┌────────────┐ ┌────────────────────┐ │
┌─────────────────────┐ WSS audio + control │ │ REST/UI │ │ Device hub │ │
│ Pi client (Python) │ ◀───────────────────────────▶│ │ + Identity│ │ (per-device WS) │ │
│ wakeword + audio │ │ └────────────┘ └─────────┬──────────┘ │
└─────────────────────┘ │ │ │
│ ┌────────────┐ ┌────────▼──────────┐ │
│ │ Tools │ │ Realtime relay │ │
│ │ registry │◀─┤ (per-session WS │ │
│ └────────────┘ │ upstream) │ │
│ └─────────┬──────────┘ │
│ SQLite (named volume) │ │
└────────────────────────────┼────────────┘
│ WSS
▼
OpenAI Realtime API
```
One container; React SPA served by the same ASP.NET process. SQLite file lives on a Coolify named volume. The OpenAI API key is held server-side; clients never see it.
### Why backend relay (not direct Pi ↔ OpenAI)
- One API key, server-side only.
- All user-side and assistant-side transcripts land in our database directly off the upstream WS events (no separate STT pass).
- Tool definitions and changes don't require a client redeploy.
- Backend can push audio to a Pi on its own (future feature) over the same WS.
### Backend
Single ASP.NET 9 project (`backend/`) organised into modules:
```
backend/
├── Program.cs # composition root, EF migrate, route + hub mapping
├── Auth/ # ASP.NET Identity, roles (Admin, User), invite flow
├── Devices/ # device CRUD, pairing-code issuance, token mgmt
├── DeviceHub/ # WebSocket handler at /device, per-device session state
├── Realtime/ # RealtimeRelay: per-conversation upstream WS to OpenAI
├── Tools/ # ITool interface, registry, built-in tools
├── Conversations/ # persistence of turns, transcripts, tool calls
├── Install/ # serves install.sh and client.tar.gz
├── Data/ # EF Core DbContext, migrations, SQLite
└── wwwroot/ # built React SPA (Vite output) + client.tar.gz
```
Key services (DI):
- `DeviceRegistry` (singleton) — in-memory map `deviceId → ActiveDevice`, tracks online state and last frame timestamp.
- `RealtimeSessionFactory` (singleton) — given a `DeviceContext`, opens an upstream WSS to OpenAI and returns a `RealtimeSession` instance bound to one conversation.
- `ToolRegistry` (singleton) — static list of `ITool`s registered at startup, filterable by per-device enabled set.
- `ConversationLog` (scoped) — append-only writer for turns; flushes per event so a crash mid-conversation still preserves what was said.
- `OpenAIKeyProvider` (singleton) — reads `OPENAI_API_KEY` from env.
### Pi client
Python package (`client/`), deployed to `~/assistant/code/` on the Pi:
```
client/
├── main.py # entry: sets PA_ALSA_PLUGHW BEFORE importing sounddevice
├── audio.py # USB device discovery, persistent in/out streams
├── wakeword.py # openwakeword wrapper + gating flag
├── session.py # backend WS client: binary audio + JSON control
├── playback.py # queue-fed playback, wake/sleep wav hooks
├── state.py # state machine
├── config.py # loads ~/assistant/state/config.json
├── pair.py # one-shot: prompt for code, hit /api/pair, write config.json
└── log.py # structured logging to journald
```
State persists at `~/assistant/state/`:
- `config.json` (mode 0600) — `device_id`, `device_token`, `backend_ws`
- `wake.wav` / `sleep.wav` — optional cues, played if present
- (room for future per-device assets; never rsynced/overwritten by deploys)
`~/assistant/code/` is the only directory that the install script replaces on update; `~/assistant/state/` is untouched.
### State machine
```
wakeword fires session_started
┌─────────────────────┐ ┌──────────────────────┐
│ ▼ │ ▼
[IDLE] ─────────► [WAKE_PENDING] ─────► [LISTENING] ───────► [ASSISTANT_SPEAKING]
▲ │ ▲ │
│ │ │ assistant done │
│ └───┴──────────────────┘
│ session_ended
└─────────────────────────────── (idle timeout OR end_session tool)
plays sleep.wav, re-enables wakeword
```
- **IDLE** — wakeword detector enabled, no upstream audio. Persistent backend WS open with heartbeats.
- **WAKE_PENDING** — wakeword fired; play `wake.wav` (non-blocking); send `wake`; wait for `session_started`. Wakeword disabled.
- **LISTENING** — mic frames stream to backend; server-VAD on OpenAI decides turn boundaries; we don't compute local VAD.
- **ASSISTANT_SPEAKING** — uplink muted; inbound frames flow to playback queue.
- On `assistant_done` from backend → LISTENING again (uplink unmuted).
- On `session_ended` from backend → wait for current audio to drain, play `sleep.wav`, send `playback_done(for=sleep)` → IDLE.
### Audio plumbing
- **Wire format**: PCM16 little-endian, 24 kHz mono, exactly 1920 samples (3840 bytes, 80 ms) per binary frame. Matches OpenAI Realtime's default `pcm16`. No transcoding in the backend.
- **Capture**: single InputStream, callback-driven, blocksize 1920, sample rate 24 000 Hz. Callback only enqueues — never blocks.
- **Playback**: single OutputStream opened at startup, persistent. Playback worker pulls from an `asyncio.Queue` and writes — never `sd.play()/sd.wait()` (which would block the InputStream the way the prototype did).
- **Wakeword feed**: each captured 1920-sample frame is downsampled to 1280 samples (80 ms at 16 kHz) via `scipy.signal.resample_poly(frame, up=2, down=3)` and fed to `model.predict()`.
- `PA_ALSA_PLUGHW=1` is set at the very top of `main.py` before any `sounddevice` import.
### Threading model
- Main thread runs an asyncio loop owning the backend WS.
- PortAudio invokes the InputStream callback on its own thread; the callback writes mic frames to a bounded `queue.Queue`. On overflow, oldest frame is dropped and a warning is logged.
- A bridge thread reads `queue.Queue`, runs wakeword.predict() when state is IDLE/WAKE_PENDING, and forwards bytes to the asyncio loop via `loop.call_soon_threadsafe` for upstream sends when state is LISTENING.
- A playback worker (asyncio task) drains an `asyncio.Queue` and writes to the OutputStream in a thread executor.
### Reconnect & heartbeat
- Pi → backend WS reconnects with exponential backoff 1 → 30 s.
- While disconnected, state stays IDLE; wakewords are ignored.
- An in-flight conversation does **not** survive a network drop — backend marks conversation `EndReason="error"`.
- Client sends `{"type":"ping"}` every 15 s when IDLE; backend uses it to update `Devices.LastSeenAt` and drive the UI online indicator.
## Wire protocol (Pi ↔ backend)
One persistent WebSocket at `wss://<backend>/device`. Authenticated via `Authorization: Bearer <device-token>` on the upgrade request; connection rejected if the token is revoked or unknown.
### Binary frames
Raw PCM16 LE, 24 kHz mono, exactly 1920 samples (3840 bytes). No header. Direction inferred from sender. Backend drops binary frames received outside an active session. Wrong-sized frames are dropped silently with a log entry (protects against runaways without surfacing an error).
### Text frames (JSON envelopes)
Client → backend:
| `type` | Fields | Notes |
| ----------------- | ----------------------------------- | ----------------------------------------------------------- |
| `hello` | `client_version`, `device_id` | Sent on connect; backend replies `hello_ack` with the device's config snapshot. |
| `ping` | — | Every 15 s while IDLE. Backend replies `pong`. |
| `wake` | `at` (client ms timestamp) | Triggers backend to open upstream Realtime WS. |
| `cancel` | — | User-initiated abort (e.g. SIGINT). Backend closes session. |
| `playback_done` | `for` (`wake` \| `sleep` \| `assistant_turn`) | Lets backend know when audio finished playing on the Pi (needed to time `session_ended` after `sleep.wav`). |
| `tool_result` | `call_id`, `ok`, `result?`, `error?` | Reply to a Pi-side tool call (e.g. `set_volume`). |
Backend → client:
| `type` | Fields | Notes |
| ------------------ | ----------------------------------- | ---------------------------------------------------------- |
| `hello_ack` | `config` (voice/model/prompt/tools/idle_timeout) | Client uses this to set local state. |
| `pong` | — | |
| `config_updated` | `config` | Pushed when admin saves changes in the UI; takes effect next session. |
| `session_started` | `conversation_id` | Realtime upstream is ready; client may start uplink. |
| `assistant_done` | — | Assistant turn ended; client unmutes uplink. |
| `session_ended` | `reason` (`tool` \| `idle` \| `error`) | Client plays `sleep.wav` (if present), then `playback_done(for=sleep)`. |
| `tool_call` | `call_id`, `name`, `arguments` | Invoked for tools whose execution requires the Pi. |
| `play_audio` | `kind` (`session` \| `notification`) | Marker; subsequent binary frames are payload. Ended by `play_audio_end`. |
| `play_audio_end` | — | End-of-payload marker for `play_audio(kind=notification)`. |
| `error` | `code`, `message`, `fatal?` | Logged on client; non-fatal unless `fatal=true`. |
Uplink muting is purely a client-side decision based on state. Backend doesn't enforce it.
## Conversation lifecycle
```
PI BACKEND OPENAI REALTIME
─────────────────────────────────────────────────────────────────────────────────
state=IDLE
mic frames → wakeword.predict
wakeword fires
play wake.wav (non-blocking)
send {type:"wake"} ──►
open WSS to OpenAI ──►
send session.update ──►
(voice, instructions,
input_audio_transcription,
tools=[end_session,
get_current_time,
set_volume])
◄── session.updated
insert Conversations row
send {type:"session_started",
conversation_id} ──►
state=LISTENING
mic frames (binary) ──────────────► forward as input_audio_buffer.append ─►
◄── input_audio_buffer.speech_started
◄── input_audio_buffer.speech_stopped
◄── response.created
◄── response.audio.delta (x N)
binary frames ◄─────────────── forward to PI as binary
binary frames ──► playback queue
state=ASSISTANT_SPEAKING
(uplink muted)
◄── response.audio_transcript.done
(persist assistant Turn)
◄── response.done
send {type:"assistant_done"} ──►
state=LISTENING
◄── conversation.item.input_audio_transcription.completed
(persist user Turn)
...follow-up turn(s)...
Path A — model calls end_session
◄── response.function_call_arguments.done
(name="end_session")
◄── response.audio.delta (the "bye" reply)
forward audio to PI ──────────►
wait for response.done
◄── response.done
send {type:"assistant_done"} ──►
send {type:"session_ended",
reason="tool"} ──►
close upstream WSS
persist Conversations.EndReason
state=IDLE-PENDING
wait for current audio buffer drain
play sleep.wav
send {type:"playback_done",
for:"sleep"} ──►
state=IDLE
(wakeword re-enabled)
Path B — idle timeout
no mic activity for IdleTimeoutSeconds while LISTENING
send {type:"session_ended",
reason="idle"} ──►
close upstream WSS
(same sleep.wav + transition as Path A)
```
### `end_session` sequencing
The tool call arrives *during* the assistant's response. The relay must:
1. Forward all audio deltas until `response.done`.
2. Then send `assistant_done`, then `session_ended(reason=tool)`.
3. NOT execute the tool mid-response — `end_session` has no real side effect; the relay records "the model asked to end" and closes after the audio finishes. The tool result returned to OpenAI is `{"ok": true}` (for cleanliness; the upstream is about to close).
## Tools registry
C# interface, registered as singletons at startup:
```csharp
public interface ITool
{
string Name { get; } // OpenAI tool name
string Description { get; } // shown to the model
JsonSchema ParameterSchema { get; } // OpenAI-compatible
Task<ToolResult> ExecuteAsync(
ToolInvocation invocation,
DeviceContext device,
CancellationToken ct);
}
public record ToolInvocation(string CallId, JsonElement Arguments);
public record ToolResult(JsonElement Output); // Output is JSON sent back to model
public record DeviceContext(Guid DeviceId, Guid ConversationId, IDeviceChannel Channel);
```
`IDeviceChannel` lets a tool send `tool_call`-style messages to the Pi and await `tool_result`. Tools that don't need the Pi simply don't use it.
### v1 built-in tools
1. **`end_session`** — no parameters. Sets a "model wants to end" flag on the active `RealtimeSession`; the relay's `response.done` handler closes the upstream WS and emits `session_ended(reason=tool)`. Returns `{"ok": true}`.
2. **`get_current_time`** — no parameters. Returns `{"now": "<ISO-8601 UTC>"}` from `DateTimeOffset.UtcNow`. Smoke test for the tool plane.
3. **`set_volume`** — `level: integer 0..100`. Backend sends `tool_call(name=set_volume, arguments={level})` to the Pi; Pi runs `amixer set 'PCM' N%` (or device-specific mixer) and replies with `tool_result(call_id, ok, error?)`. Backend forwards back to OpenAI as the function call output.
The per-device enabled set is enforced by filtering the tool list in `session.update`. Disabled tools are simply not advertised to the model.
## Data model (SQLite via EF Core)
```
AspNetUsers, AspNetRoles, ... # Identity standard
Devices
Id (Guid) OwnerUserId Name PairedAt LastSeenAt TokenHash IsRevoked
DeviceConfigs # 1:1 with Devices
DeviceId SystemPrompt Voice Model IdleTimeoutSeconds EnabledToolsJson
PairingCodes
Code (8 chars) OwnerUserId ExpiresAt ConsumedAt
Conversations
Id DeviceId StartedAt EndedAt EndReason ("tool" | "idle" | "error")
Turns
Id ConversationId Index Role ("user" | "assistant" | "tool") Text ToolName ToolArgsJson ToolResultJson CreatedAt
SystemSettings # singleton row
DefaultVoice DefaultModel DefaultSystemPrompt
```
- `TokenHash` = SHA-256 of the device token; plaintext is shown once during pairing and never persisted.
- `EnabledToolsJson` is a small JSON array of tool names; no relational table needed.
- `Turns.Index` is monotonic per-conversation for ordering.
- `Database.Migrate()` runs on startup; SQLite file at `/data/assistant.db` on a Coolify named volume.
## Pairing flow
UI side:
1. User signs in, clicks **Add device**, enters a friendly name.
2. Backend generates an 8-char alphanumeric code from a 32-char alphabet (no `0/O/1/l/I`); stores `PairingCodes(Code, OwnerUserId, ExpiresAt = NOW + 10 min)`.
3. UI shows the code and the one-line installer:
```
curl -fsSL https://<backend>/install.sh | bash
```
4. Code stays visible on the device page until consumed or expired; regenerate at any time.
Pi side, first run:
1. Installer runs.
2. If `~/assistant/state/config.json` exists with a valid `device_token`, **skip pairing** and proceed to update path.
3. Otherwise prompt `Pairing code: `.
4. `POST https://<backend>/api/pair` with `{ "code", "hostname", "client_version" }`.
5. Backend validates code (exists, unexpired, unconsumed), creates `Devices(OwnerUserId = code.OwnerUserId, Name = <user-set-name>, TokenHash = sha256(newToken))`, marks code `ConsumedAt = NOW`, creates default `DeviceConfigs` row from `SystemSettings`, returns `{ "device_id", "device_token", "backend_ws" }`.
6. Installer writes `~/assistant/state/config.json` (mode 0600).
Security:
- Pairing codes are one-shot, 10-min TTL, 8 chars from a 32-char alphabet (~40 bits — fine for the TTL window).
- Device tokens are 32 bytes random, base64url-encoded, stored only as SHA-256 hash server-side.
## Install script
`install.sh` is served by the backend at `GET /install.sh` and is fully idempotent.
```
1. Resolve BACKEND_URL from the script's own served-from origin (no hardcoding).
2. apt-get install -y python3-venv libportaudio2 alsa-utils
(sudo prompt OK; passwordless-sudo per findings.md §E is the recommended setup).
3. mkdir -p ~/assistant/{code,state}
4. python3 -m venv ~/assistant/.venv (if missing)
5. curl -fsSL $BACKEND_URL/client.tar.gz | tar -xz -C ~/assistant/code
6. ~/assistant/.venv/bin/pip install -r ~/assistant/code/requirements.txt
7. ~/assistant/.venv/bin/pip install --no-deps -r ~/assistant/code/requirements-nodeps.txt
(openwakeword install split per findings.md §3).
8. If config.json missing → run `~/assistant/.venv/bin/python -m client.pair`,
which prompts for the code and writes config.json.
9. Install systemd user unit at ~/.config/systemd/user/assistant.service.
10. systemctl --user daemon-reload
11. systemctl --user enable --now assistant
12. loginctl enable-linger pi (so the unit runs without an active login).
13. Print status; if pairing happened, print "Paired to <backend> as device <name>".
```
The systemd user unit runs `~/assistant/.venv/bin/python -m client.main` with:
- `Restart=on-failure`
- `RestartSec=5`
- `Environment=PA_ALSA_PLUGHW=1`
### Updates
Re-running the same one-liner upgrades the client. Steps 5–7 fetch and reinstall the latest bundle; step 11 restarts the unit. Pairing is skipped because `config.json` already exists.
`curl ... | bash -s -- --reset` deletes `config.json` and forces a fresh pairing.
### Backend install endpoints
- `GET /install.sh` — returns the shell script with `BACKEND_URL` baked from the request's `X-Forwarded-Host` / `Host`. Public, no auth.
- `GET /client.tar.gz` — returns the current client bundle (built at deploy time into `wwwroot/`). Public, no auth — the code is not a secret; the device token gates the device WS.
- `POST /api/pair` — accepts pairing code, returns device token. No prior auth (code is the credential).
The bundle is built during the Docker image build: a stage runs `tar czf client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt` and copies the result into `wwwroot/`.
## Error handling
| Scenario | Detection | Backend action | Client action |
| --- | --- | --- | --- |
| Pi loses network | WS close / heartbeat miss | Close upstream Realtime WS if active, log conversation `EndReason="error"`, mark device offline | Exponential backoff reconnect 1→30 s, stay IDLE, no wakeword |
| Backend restarts | Pi WS closes from server side | (n/a — process gone) | Same backoff reconnect; on reconnect send fresh `hello` |
| OpenAI WS drops mid-conversation | upstream `on_close` | Send `error(code=upstream)` + `session_ended(reason=error)`, persist partial transcript | Play `sleep.wav`, return to IDLE |
| OpenAI rate limit / 429 on connect | upstream open fails | Don't open session; send `error(code=upstream, message=...)` instead of `session_started` | Play `sleep.wav`, IDLE |
| Tool execution exception (server-side) | C# try/catch around `ITool.ExecuteAsync` | Send tool result `{"error": "..."}` back to OpenAI so the model can recover; log | (no client involvement) |
| Pi-side tool fails (e.g. `set_volume` amixer error) | `tool_result.ok=false` | Forward `{"error": "..."}` to OpenAI | Log and continue |
| Device token revoked while connected | Admin clicks revoke | Set `IsRevoked=true`, close all WS for that device with `error(code=auth_revoked, fatal=true)` | Stop the process (systemd will restart, fail to auth, sleep 60 s, retry — visible in logs) |
| SQLite write failure | EF exception | Bubble up to REST as 500; on the hub side, drop the offending log event, keep session running | (n/a) |
| Two simultaneous wakeword fires | `wake` while not IDLE | Backend ignores second `wake` (idempotent within active session) | Client also gates locally |
| Audio buffer overflow on Pi | PortAudio callback queue full | (n/a) | Drop oldest frame, log a warning |
## Management UI (v1)
React SPA, served from the backend's `wwwroot/`. Pages:
- **Sign in / sign up** (ASP.NET Identity).
- **Devices** — list of the signed-in user's devices: name, online indicator, last-seen.
- **Device detail** — rename; generate pairing code (with countdown); revoke; per-device config form: system prompt override, voice, model, idle timeout slider, enabled-tools checkboxes; conversation history viewer (paginated, text only).
- **Admin** (admin role only) — user list, invite/disable users, global model/voice/prompt defaults.
Out of v1: live conversation tail, audio meters, audio recordings, memory UI, per-device `wake.wav`/`sleep.wav` upload (filesystem only for v1; v1.1 adds upload).
## Testing strategy
- **Backend unit tests** (xUnit): `ToolRegistry` filtering by enabled set; `PairingService` (code generation, expiry, single-use); `ConversationLog` (turn ordering, persistence); auth/role enforcement on the admin endpoints.
- **Backend integration tests** (`WebApplicationFactory`): full pair flow end-to-end against in-memory SQLite; device-hub WS with a fake OpenAI upstream (test double that emits scripted Realtime events).
- **Client unit tests** (pytest): state machine transitions (pure-Python, no audio); resample correctness against golden samples; config loading. Audio I/O and wakeword are not unit-tested — already covered by the prototypes and by manual end-to-end.
- **End-to-end manual smoke test** (in repo `README.md`): pair a fresh Pi, say wakeword, ask "what time is it", say "bye", verify conversation row + sleep.wav heard. Run after any change that touches the conversation flow.
- No load / perf testing.
## Deployment (Coolify)
Single Docker image, multi-stage Dockerfile (`preparing-dotnet-react-app-for-coolify` skill is the reference for the layout):
1. **Frontend build** stage — Node, `npm ci && npm run build` against `frontend/`.
2. **Backend build** stage — .NET SDK, `dotnet publish backend/ -c Release -o /publish`.
3. **Client bundle** stage — `tar czf /publish/wwwroot/client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt`.
4. **Final** stage — `mcr.microsoft.com/dotnet/aspnet:9.0`, copies `/publish`, copies SPA `dist/` into `wwwroot/`, sets `ASPNETCORE_URLS=http://+:8080`, exposes 8080.
Coolify app (provisioned via `deploying-to-coolify-via-api`):
- Source: this repo on Gitea.
- Expose 8080 (Traefik fronts it).
- `SERVICE_FQDN_BACKEND` → backend domain.
- Env vars:
- `OPENAI_API_KEY` — secret.
- `ConnectionStrings__Default=Data Source=/data/assistant.db`
- `ASPNETCORE_URLS=http://+:8080`
- Named volumes:
- `/data` — SQLite file.
- `/keys` — ASP.NET data-protection keys (cookies survive container restarts).
- Deploys triggered fire-and-forget via the Coolify v4 API (per global CLAUDE.md rule).
## Repo layout (final)
```
Assistant/
├── CLAUDE.md
├── README.md # smoke-test instructions
├── findings.md # prototype-phase findings (kept)
├── tests/ # prototype throwaways (kept for reference)
│ ├── 01-record-play/
│ └── 02-wakeword/
├── docs/superpowers/
│ ├── specs/
│ │ ├── 2026-06-11-voice-assistant-prototypes-design.md
│ │ └── 2026-06-11-smart-assistant-design.md # this design
│ └── plans/
│ └── (one or more implementation plans, written next)
├── backend/ # ASP.NET solution
│ ├── backend.csproj
│ ├── Program.cs
│ ├── Auth/ Devices/ DeviceHub/ Realtime/ Tools/ Conversations/ Install/ Data/
│ └── wwwroot/ # filled at build: SPA dist + client.tar.gz
├── frontend/ # React + Vite source
│ ├── package.json
│ └── src/
├── client/ # Python package
│ ├── requirements.txt # sounddevice, numpy, scipy, onnxruntime, websockets, ...
│ ├── requirements-nodeps.txt # openwakeword
│ ├── main.py audio.py wakeword.py session.py playback.py state.py config.py pair.py log.py
│ └── systemd/assistant.service
├── Dockerfile
└── deploy.json # Coolify config (project/server uuids, domain)
```
## Open items deferred to implementation plans
- Exact OpenAI Realtime model + voice default (`SystemSettings.DefaultModel`/`DefaultVoice`) — pick at implementation time from currently-available choices.
- Whether wake.wav / sleep.wav playback uses the persistent OutputStream (preferred — same path as session audio) or a separate one-shot — settled in the client plan.
- ASP.NET data-protection key seeding (initial bootstrap on first run).
- React UI component library — Tailwind only, or Tailwind + a small headless component lib (Radix / shadcn).
- Exact mixer control name for `set_volume` on the LISTENAI device — confirm with `amixer scontrols` on the Pi during implementation.
## Reference paths
- Prototype spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`
- Prototype findings: `findings.md`
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
- Coolify skills: `deploying-to-coolify-via-api`, `preparing-dotnet-react-app-for-coolify`