tes/Assistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
Assistant/docs/superpowers/specs/2026-06-11-smart-assistant-design.md
T
tes 99a9950cee Smart-assistant main-build design spec 
Captures the architecture for the post-prototype build: Pi Python client +
ASP.NET backend (with embedded React UI) on Coolify, SQLite, multi-turn
Realtime sessions with end_session/get_current_time/set_volume tools,
pairing-code device onboarding, and idempotent install/update flow.
2026-06-11 15:48:10 +00:00

517 lines
33 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Smart Assistant — design
**Date:** 2026-06-11
**Status:** approved (brainstorming complete)
**Scope:** main build of the voice assistant, replacing the throwaway prototypes in `tests/`. Two deliverables: a Python client on the Raspberry Pi, and an ASP.NET backend (with embedded React management UI) deployed to Coolify. Brings up everything needed to wake on a wakeword, hold a multi-turn conversation through the OpenAI Realtime API, persist a text history per device, run a small built-in tool registry, and let users manage their devices through a browser.
## Context
The prototype phase (`docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`, results in `findings.md`) proved that recording/playback and openwakeword both work on the target Pi. The main assistant is built on top of those findings.
Key carry-forwards from the prototypes:
- USB Speaker Phone (ALSA card 3, Anhui LISTENAI) is the only mic/speaker. 48 kHz native; `PA_ALSA_PLUGHW=1` set before importing `sounddevice` lets us open at any rate.
- `python-sounddevice` + `numpy` are the audio stack.
- `openwakeword` runs at ~0.6 s load, well under 50% of one core; the `alexa` stock model is reused for v1 (custom-trained wakeword is a separate follow-up task).
- `openwakeword==0.6.0` needs `--no-deps` install on Python 3.13 because its `tflite-runtime` pin has no wheel; the prototype's two-pass install pattern is kept.
- Speaker hears its own output → detector must be gated while playing audio, and uplink to OpenAI must be muted while the assistant is speaking.
- TTS/beep must not block the input stream — callback-driven InputStream + persistent OutputStream is the pattern.
## Goals
1. A fresh Raspberry Pi can be paired to a user account by running one curl-piped install command and entering a short code.
2. The Pi wakes on the wakeword, opens a Realtime conversation, supports multi-turn until the user says "bye" or goes idle, and plays optional `wake.wav` / `sleep.wav` cues.
3. A signed-in user sees only their own devices and conversations in the management UI.
4. An admin user can manage system-wide defaults and other users.
5. Per-device configuration: name, system prompt override, voice, model, idle timeout, enabled-tools toggles. Changes take effect on the next session.
6. Text-only conversation history per device, viewable in the UI.
7. Built-in C# tool registry with `end_session`, `get_current_time`, `set_volume` shipped in v1.
8. Backend deployed to Coolify as one container with one domain; deploys triggered fire-and-forget via the Coolify v4 API.
9. The install script is the update mechanism: re-running it upgrades the client in place without re-pairing.
## Non-goals (v1)
- Long-term memory across sessions.
- MCP-based external tools (only built-in C# tools).
- Audio recordings of conversations (text only).
- Live conversation tailing or audio meters in the UI.
- Per-device upload of `wake.wav`/`sleep.wav` via UI (filesystem only — v1.1).
- Custom-trained wakeword (uses the stock `alexa` model for v1).
- Barge-in (interrupting the assistant mid-reply).
- Multi-language (English only).
- Offline / local-only fallback when the network is down.
- OIDC sign-in (local ASP.NET Identity only).
- `.deb` packaging for the client.
- Load / performance testing.
## Architecture
### Topology
```
┌─────────────────────┐ pair code / token ┌──────────────────────────────────────────┐
│ Browser (admin UI) │ ──────HTTPS (REST)────────▶ │ │
│ React SPA │ │ ASP.NET backend (Coolify container) │
└─────────────────────┘ │ │
│ ┌────────────┐ ┌────────────────────┐ │
┌─────────────────────┐ WSS audio + control │ │ REST/UI │ │ Device hub │ │
│ Pi client (Python) │ ◀───────────────────────────▶│ │ + Identity│ │ (per-device WS) │ │
│ wakeword + audio │ │ └────────────┘ └─────────┬──────────┘ │
└─────────────────────┘ │ │ │
│ ┌────────────┐ ┌────────▼──────────┐ │
│ │ Tools │ │ Realtime relay │ │
│ │ registry │◀─┤ (per-session WS │ │
│ └────────────┘ │ upstream) │ │
│ └─────────┬──────────┘ │
│ SQLite (named volume) │ │
└────────────────────────────┼────────────┘
│ WSS
▼
OpenAI Realtime API
```
One container; React SPA served by the same ASP.NET process. SQLite file lives on a Coolify named volume. The OpenAI API key is held server-side; clients never see it.
### Why backend relay (not direct Pi ↔ OpenAI)
- One API key, server-side only.
- All user-side and assistant-side transcripts land in our database directly off the upstream WS events (no separate STT pass).
- Tool definitions and changes don't require a client redeploy.
- Backend can push audio to a Pi on its own (future feature) over the same WS.
### Backend
Single ASP.NET 9 project (`backend/`) organised into modules:
```
backend/
├── Program.cs # composition root, EF migrate, route + hub mapping
├── Auth/ # ASP.NET Identity, roles (Admin, User), invite flow
├── Devices/ # device CRUD, pairing-code issuance, token mgmt
├── DeviceHub/ # WebSocket handler at /device, per-device session state
├── Realtime/ # RealtimeRelay: per-conversation upstream WS to OpenAI
├── Tools/ # ITool interface, registry, built-in tools
├── Conversations/ # persistence of turns, transcripts, tool calls
├── Install/ # serves install.sh and client.tar.gz
├── Data/ # EF Core DbContext, migrations, SQLite
└── wwwroot/ # built React SPA (Vite output) + client.tar.gz
```
Key services (DI):
- `DeviceRegistry` (singleton) — in-memory map `deviceId → ActiveDevice`, tracks online state and last frame timestamp.
- `RealtimeSessionFactory` (singleton) — given a `DeviceContext`, opens an upstream WSS to OpenAI and returns a `RealtimeSession` instance bound to one conversation.
- `ToolRegistry` (singleton) — static list of `ITool`s registered at startup, filterable by per-device enabled set.
- `ConversationLog` (scoped) — append-only writer for turns; flushes per event so a crash mid-conversation still preserves what was said.
- `OpenAIKeyProvider` (singleton) — reads `OPENAI_API_KEY` from env.
### Pi client
Python package (`client/`), deployed to `~/assistant/code/` on the Pi:
```
client/
├── main.py # entry: sets PA_ALSA_PLUGHW BEFORE importing sounddevice
├── audio.py # USB device discovery, persistent in/out streams
├── wakeword.py # openwakeword wrapper + gating flag
├── session.py # backend WS client: binary audio + JSON control
├── playback.py # queue-fed playback, wake/sleep wav hooks
├── state.py # state machine
├── config.py # loads ~/assistant/state/config.json
├── pair.py # one-shot: prompt for code, hit /api/pair, write config.json
└── log.py # structured logging to journald
```
State persists at `~/assistant/state/`:
- `config.json` (mode 0600) — `device_id`, `device_token`, `backend_ws`
- `wake.wav` / `sleep.wav` — optional cues, played if present
- (room for future per-device assets; never rsynced/overwritten by deploys)
`~/assistant/code/` is the only directory that the install script replaces on update; `~/assistant/state/` is untouched.
### State machine
```
wakeword fires session_started
┌─────────────────────┐ ┌──────────────────────┐
│ ▼ │ ▼
[IDLE] ─────────► [WAKE_PENDING] ─────► [LISTENING] ───────► [ASSISTANT_SPEAKING]
▲ │ ▲ │
│ │ │ assistant done │
│ └───┴──────────────────┘
│ session_ended
└─────────────────────────────── (idle timeout OR end_session tool)
plays sleep.wav, re-enables wakeword
```
- **IDLE** — wakeword detector enabled, no upstream audio. Persistent backend WS open with heartbeats.
- **WAKE_PENDING** — wakeword fired; play `wake.wav` (non-blocking); send `wake`; wait for `session_started`. Wakeword disabled.
- **LISTENING** — mic frames stream to backend; server-VAD on OpenAI decides turn boundaries; we don't compute local VAD.
- **ASSISTANT_SPEAKING** — uplink muted; inbound frames flow to playback queue.
- On `assistant_done` from backend → LISTENING again (uplink unmuted).
- On `session_ended` from backend → wait for current audio to drain, play `sleep.wav`, send `playback_done(for=sleep)` → IDLE.
### Audio plumbing
- **Wire format**: PCM16 little-endian, 24 kHz mono, exactly 1920 samples (3840 bytes, 80 ms) per binary frame. Matches OpenAI Realtime's default `pcm16`. No transcoding in the backend.
- **Capture**: single InputStream, callback-driven, blocksize 1920, sample rate 24 000 Hz. Callback only enqueues — never blocks.
- **Playback**: single OutputStream opened at startup, persistent. Playback worker pulls from an `asyncio.Queue` and writes — never `sd.play()/sd.wait()` (which would block the InputStream the way the prototype did).
- **Wakeword feed**: each captured 1920-sample frame is downsampled to 1280 samples (80 ms at 16 kHz) via `scipy.signal.resample_poly(frame, up=2, down=3)` and fed to `model.predict()`.
- `PA_ALSA_PLUGHW=1` is set at the very top of `main.py` before any `sounddevice` import.
### Threading model
- Main thread runs an asyncio loop owning the backend WS.
- PortAudio invokes the InputStream callback on its own thread; the callback writes mic frames to a bounded `queue.Queue`. On overflow, oldest frame is dropped and a warning is logged.
- A bridge thread reads `queue.Queue`, runs wakeword.predict() when state is IDLE/WAKE_PENDING, and forwards bytes to the asyncio loop via `loop.call_soon_threadsafe` for upstream sends when state is LISTENING.
- A playback worker (asyncio task) drains an `asyncio.Queue` and writes to the OutputStream in a thread executor.
### Reconnect & heartbeat
- Pi → backend WS reconnects with exponential backoff 1 → 30 s.
- While disconnected, state stays IDLE; wakewords are ignored.
- An in-flight conversation does **not** survive a network drop — backend marks conversation `EndReason="error"`.
- Client sends `{"type":"ping"}` every 15 s when IDLE; backend uses it to update `Devices.LastSeenAt` and drive the UI online indicator.
## Wire protocol (Pi ↔ backend)
One persistent WebSocket at `wss://<backend>/device`. Authenticated via `Authorization: Bearer <device-token>` on the upgrade request; connection rejected if the token is revoked or unknown.
### Binary frames
Raw PCM16 LE, 24 kHz mono, exactly 1920 samples (3840 bytes). No header. Direction inferred from sender. Backend drops binary frames received outside an active session. Wrong-sized frames are dropped silently with a log entry (protects against runaways without surfacing an error).
### Text frames (JSON envelopes)
Client → backend:
| `type` | Fields | Notes |
| ----------------- | ----------------------------------- | ----------------------------------------------------------- |
| `hello` | `client_version`, `device_id` | Sent on connect; backend replies `hello_ack` with the device's config snapshot. |
| `ping` | — | Every 15 s while IDLE. Backend replies `pong`. |
| `wake` | `at` (client ms timestamp) | Triggers backend to open upstream Realtime WS. |
| `cancel` | — | User-initiated abort (e.g. SIGINT). Backend closes session. |
| `playback_done` | `for` (`wake` \| `sleep` \| `assistant_turn`) | Lets backend know when audio finished playing on the Pi (needed to time `session_ended` after `sleep.wav`). |
| `tool_result` | `call_id`, `ok`, `result?`, `error?` | Reply to a Pi-side tool call (e.g. `set_volume`). |
Backend → client:
| `type` | Fields | Notes |
| ------------------ | ----------------------------------- | ---------------------------------------------------------- |
| `hello_ack` | `config` (voice/model/prompt/tools/idle_timeout) | Client uses this to set local state. |
| `pong` | — | |
| `config_updated` | `config` | Pushed when admin saves changes in the UI; takes effect next session. |
| `session_started` | `conversation_id` | Realtime upstream is ready; client may start uplink. |
| `assistant_done` | — | Assistant turn ended; client unmutes uplink. |
| `session_ended` | `reason` (`tool` \| `idle` \| `error`) | Client plays `sleep.wav` (if present), then `playback_done(for=sleep)`. |
| `tool_call` | `call_id`, `name`, `arguments` | Invoked for tools whose execution requires the Pi. |
| `play_audio` | `kind` (`session` \| `notification`) | Marker; subsequent binary frames are payload. Ended by `play_audio_end`. |
| `play_audio_end` | — | End-of-payload marker for `play_audio(kind=notification)`. |
| `error` | `code`, `message`, `fatal?` | Logged on client; non-fatal unless `fatal=true`. |
Uplink muting is purely a client-side decision based on state. Backend doesn't enforce it.
## Conversation lifecycle
```
PI BACKEND OPENAI REALTIME
─────────────────────────────────────────────────────────────────────────────────
state=IDLE
mic frames → wakeword.predict
wakeword fires
play wake.wav (non-blocking)
send {type:"wake"} ──►
open WSS to OpenAI ──►
send session.update ──►
(voice, instructions,
input_audio_transcription,
tools=[end_session,
get_current_time,
set_volume])
◄── session.updated
insert Conversations row
send {type:"session_started",
conversation_id} ──►
state=LISTENING
mic frames (binary) ──────────────► forward as input_audio_buffer.append ─►
◄── input_audio_buffer.speech_started
◄── input_audio_buffer.speech_stopped
◄── response.created
◄── response.audio.delta (x N)
binary frames ◄─────────────── forward to PI as binary
binary frames ──► playback queue
state=ASSISTANT_SPEAKING
(uplink muted)
◄── response.audio_transcript.done
(persist assistant Turn)
◄── response.done
send {type:"assistant_done"} ──►
state=LISTENING
◄── conversation.item.input_audio_transcription.completed
(persist user Turn)
...follow-up turn(s)...
Path A — model calls end_session
◄── response.function_call_arguments.done
(name="end_session")
◄── response.audio.delta (the "bye" reply)
forward audio to PI ──────────►
wait for response.done
◄── response.done
send {type:"assistant_done"} ──►
send {type:"session_ended",
reason="tool"} ──►
close upstream WSS
persist Conversations.EndReason
state=IDLE-PENDING
wait for current audio buffer drain
play sleep.wav
send {type:"playback_done",
for:"sleep"} ──►
state=IDLE
(wakeword re-enabled)
Path B — idle timeout
no mic activity for IdleTimeoutSeconds while LISTENING
send {type:"session_ended",
reason="idle"} ──►
close upstream WSS
(same sleep.wav + transition as Path A)
```
### `end_session` sequencing
The tool call arrives *during* the assistant's response. The relay must:
1. Forward all audio deltas until `response.done`.
2. Then send `assistant_done`, then `session_ended(reason=tool)`.
3. NOT execute the tool mid-response — `end_session` has no real side effect; the relay records "the model asked to end" and closes after the audio finishes. The tool result returned to OpenAI is `{"ok": true}` (for cleanliness; the upstream is about to close).
## Tools registry
C# interface, registered as singletons at startup:
```csharp
public interface ITool
{
string Name { get; } // OpenAI tool name
string Description { get; } // shown to the model
JsonSchema ParameterSchema { get; } // OpenAI-compatible
Task<ToolResult> ExecuteAsync(
ToolInvocation invocation,
DeviceContext device,
CancellationToken ct);
}
public record ToolInvocation(string CallId, JsonElement Arguments);
public record ToolResult(JsonElement Output); // Output is JSON sent back to model
public record DeviceContext(Guid DeviceId, Guid ConversationId, IDeviceChannel Channel);
```
`IDeviceChannel` lets a tool send `tool_call`-style messages to the Pi and await `tool_result`. Tools that don't need the Pi simply don't use it.
### v1 built-in tools
1. **`end_session`** — no parameters. Sets a "model wants to end" flag on the active `RealtimeSession`; the relay's `response.done` handler closes the upstream WS and emits `session_ended(reason=tool)`. Returns `{"ok": true}`.
2. **`get_current_time`** — no parameters. Returns `{"now": "<ISO-8601 UTC>"}` from `DateTimeOffset.UtcNow`. Smoke test for the tool plane.
3. **`set_volume`** — `level: integer 0..100`. Backend sends `tool_call(name=set_volume, arguments={level})` to the Pi; Pi runs `amixer set 'PCM' N%` (or device-specific mixer) and replies with `tool_result(call_id, ok, error?)`. Backend forwards back to OpenAI as the function call output.
The per-device enabled set is enforced by filtering the tool list in `session.update`. Disabled tools are simply not advertised to the model.
## Data model (SQLite via EF Core)
```
AspNetUsers, AspNetRoles, ... # Identity standard
Devices
Id (Guid) OwnerUserId Name PairedAt LastSeenAt TokenHash IsRevoked
DeviceConfigs # 1:1 with Devices
DeviceId SystemPrompt Voice Model IdleTimeoutSeconds EnabledToolsJson
PairingCodes
Code (8 chars) OwnerUserId ExpiresAt ConsumedAt
Conversations
Id DeviceId StartedAt EndedAt EndReason ("tool" | "idle" | "error")
Turns
Id ConversationId Index Role ("user" | "assistant" | "tool") Text ToolName ToolArgsJson ToolResultJson CreatedAt
SystemSettings # singleton row
DefaultVoice DefaultModel DefaultSystemPrompt
```
- `TokenHash` = SHA-256 of the device token; plaintext is shown once during pairing and never persisted.
- `EnabledToolsJson` is a small JSON array of tool names; no relational table needed.
- `Turns.Index` is monotonic per-conversation for ordering.
- `Database.Migrate()` runs on startup; SQLite file at `/data/assistant.db` on a Coolify named volume.
## Pairing flow
UI side:
1. User signs in, clicks **Add device**, enters a friendly name.
2. Backend generates an 8-char alphanumeric code from a 32-char alphabet (no `0/O/1/l/I`); stores `PairingCodes(Code, OwnerUserId, ExpiresAt = NOW + 10 min)`.
3. UI shows the code and the one-line installer:
```
curl -fsSL https://<backend>/install.sh | bash
```
4. Code stays visible on the device page until consumed or expired; regenerate at any time.
Pi side, first run:
1. Installer runs.
2. If `~/assistant/state/config.json` exists with a valid `device_token`, **skip pairing** and proceed to update path.
3. Otherwise prompt `Pairing code: `.
4. `POST https://<backend>/api/pair` with `{ "code", "hostname", "client_version" }`.
5. Backend validates code (exists, unexpired, unconsumed), creates `Devices(OwnerUserId = code.OwnerUserId, Name = <user-set-name>, TokenHash = sha256(newToken))`, marks code `ConsumedAt = NOW`, creates default `DeviceConfigs` row from `SystemSettings`, returns `{ "device_id", "device_token", "backend_ws" }`.
6. Installer writes `~/assistant/state/config.json` (mode 0600).
Security:
- Pairing codes are one-shot, 10-min TTL, 8 chars from a 32-char alphabet (~40 bits — fine for the TTL window).
- Device tokens are 32 bytes random, base64url-encoded, stored only as SHA-256 hash server-side.
## Install script
`install.sh` is served by the backend at `GET /install.sh` and is fully idempotent.
```
1. Resolve BACKEND_URL from the script's own served-from origin (no hardcoding).
2. apt-get install -y python3-venv libportaudio2 alsa-utils
(sudo prompt OK; passwordless-sudo per findings.md §E is the recommended setup).
3. mkdir -p ~/assistant/{code,state}
4. python3 -m venv ~/assistant/.venv (if missing)
5. curl -fsSL $BACKEND_URL/client.tar.gz | tar -xz -C ~/assistant/code
6. ~/assistant/.venv/bin/pip install -r ~/assistant/code/requirements.txt
7. ~/assistant/.venv/bin/pip install --no-deps -r ~/assistant/code/requirements-nodeps.txt
(openwakeword install split per findings.md §3).
8. If config.json missing → run `~/assistant/.venv/bin/python -m client.pair`,
which prompts for the code and writes config.json.
9. Install systemd user unit at ~/.config/systemd/user/assistant.service.
10. systemctl --user daemon-reload
11. systemctl --user enable --now assistant
12. loginctl enable-linger pi (so the unit runs without an active login).
13. Print status; if pairing happened, print "Paired to <backend> as device <name>".
```
The systemd user unit runs `~/assistant/.venv/bin/python -m client.main` with:
- `Restart=on-failure`
- `RestartSec=5`
- `Environment=PA_ALSA_PLUGHW=1`
### Updates
Re-running the same one-liner upgrades the client. Steps 5–7 fetch and reinstall the latest bundle; step 11 restarts the unit. Pairing is skipped because `config.json` already exists.
`curl ... | bash -s -- --reset` deletes `config.json` and forces a fresh pairing.
### Backend install endpoints
- `GET /install.sh` — returns the shell script with `BACKEND_URL` baked from the request's `X-Forwarded-Host` / `Host`. Public, no auth.
- `GET /client.tar.gz` — returns the current client bundle (built at deploy time into `wwwroot/`). Public, no auth — the code is not a secret; the device token gates the device WS.
- `POST /api/pair` — accepts pairing code, returns device token. No prior auth (code is the credential).
The bundle is built during the Docker image build: a stage runs `tar czf client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt` and copies the result into `wwwroot/`.
## Error handling
| Scenario | Detection | Backend action | Client action |
| --- | --- | --- | --- |
| Pi loses network | WS close / heartbeat miss | Close upstream Realtime WS if active, log conversation `EndReason="error"`, mark device offline | Exponential backoff reconnect 1→30 s, stay IDLE, no wakeword |
| Backend restarts | Pi WS closes from server side | (n/a — process gone) | Same backoff reconnect; on reconnect send fresh `hello` |
| OpenAI WS drops mid-conversation | upstream `on_close` | Send `error(code=upstream)` + `session_ended(reason=error)`, persist partial transcript | Play `sleep.wav`, return to IDLE |
| OpenAI rate limit / 429 on connect | upstream open fails | Don't open session; send `error(code=upstream, message=...)` instead of `session_started` | Play `sleep.wav`, IDLE |
| Tool execution exception (server-side) | C# try/catch around `ITool.ExecuteAsync` | Send tool result `{"error": "..."}` back to OpenAI so the model can recover; log | (no client involvement) |
| Pi-side tool fails (e.g. `set_volume` amixer error) | `tool_result.ok=false` | Forward `{"error": "..."}` to OpenAI | Log and continue |
| Device token revoked while connected | Admin clicks revoke | Set `IsRevoked=true`, close all WS for that device with `error(code=auth_revoked, fatal=true)` | Stop the process (systemd will restart, fail to auth, sleep 60 s, retry — visible in logs) |
| SQLite write failure | EF exception | Bubble up to REST as 500; on the hub side, drop the offending log event, keep session running | (n/a) |
| Two simultaneous wakeword fires | `wake` while not IDLE | Backend ignores second `wake` (idempotent within active session) | Client also gates locally |
| Audio buffer overflow on Pi | PortAudio callback queue full | (n/a) | Drop oldest frame, log a warning |
## Management UI (v1)
React SPA, served from the backend's `wwwroot/`. Pages:
- **Sign in / sign up** (ASP.NET Identity).
- **Devices** — list of the signed-in user's devices: name, online indicator, last-seen.
- **Device detail** — rename; generate pairing code (with countdown); revoke; per-device config form: system prompt override, voice, model, idle timeout slider, enabled-tools checkboxes; conversation history viewer (paginated, text only).
- **Admin** (admin role only) — user list, invite/disable users, global model/voice/prompt defaults.
Out of v1: live conversation tail, audio meters, audio recordings, memory UI, per-device `wake.wav`/`sleep.wav` upload (filesystem only for v1; v1.1 adds upload).
## Testing strategy
- **Backend unit tests** (xUnit): `ToolRegistry` filtering by enabled set; `PairingService` (code generation, expiry, single-use); `ConversationLog` (turn ordering, persistence); auth/role enforcement on the admin endpoints.
- **Backend integration tests** (`WebApplicationFactory`): full pair flow end-to-end against in-memory SQLite; device-hub WS with a fake OpenAI upstream (test double that emits scripted Realtime events).
- **Client unit tests** (pytest): state machine transitions (pure-Python, no audio); resample correctness against golden samples; config loading. Audio I/O and wakeword are not unit-tested — already covered by the prototypes and by manual end-to-end.
- **End-to-end manual smoke test** (in repo `README.md`): pair a fresh Pi, say wakeword, ask "what time is it", say "bye", verify conversation row + sleep.wav heard. Run after any change that touches the conversation flow.
- No load / perf testing.
## Deployment (Coolify)
Single Docker image, multi-stage Dockerfile (`preparing-dotnet-react-app-for-coolify` skill is the reference for the layout):
1. **Frontend build** stage — Node, `npm ci && npm run build` against `frontend/`.
2. **Backend build** stage — .NET SDK, `dotnet publish backend/ -c Release -o /publish`.
3. **Client bundle** stage — `tar czf /publish/wwwroot/client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt`.
4. **Final** stage — `mcr.microsoft.com/dotnet/aspnet:9.0`, copies `/publish`, copies SPA `dist/` into `wwwroot/`, sets `ASPNETCORE_URLS=http://+:8080`, exposes 8080.
Coolify app (provisioned via `deploying-to-coolify-via-api`):
- Source: this repo on Gitea.
- Expose 8080 (Traefik fronts it).
- `SERVICE_FQDN_BACKEND` → backend domain.
- Env vars:
- `OPENAI_API_KEY` — secret.
- `ConnectionStrings__Default=Data Source=/data/assistant.db`
- `ASPNETCORE_URLS=http://+:8080`
- Named volumes:
- `/data` — SQLite file.
- `/keys` — ASP.NET data-protection keys (cookies survive container restarts).
- Deploys triggered fire-and-forget via the Coolify v4 API (per global CLAUDE.md rule).
## Repo layout (final)
```
Assistant/
├── CLAUDE.md
├── README.md # smoke-test instructions
├── findings.md # prototype-phase findings (kept)
├── tests/ # prototype throwaways (kept for reference)
│ ├── 01-record-play/
│ └── 02-wakeword/
├── docs/superpowers/
│ ├── specs/
│ │ ├── 2026-06-11-voice-assistant-prototypes-design.md
│ │ └── 2026-06-11-smart-assistant-design.md # this design
│ └── plans/
│ └── (one or more implementation plans, written next)
├── backend/ # ASP.NET solution
│ ├── backend.csproj
│ ├── Program.cs
│ ├── Auth/ Devices/ DeviceHub/ Realtime/ Tools/ Conversations/ Install/ Data/
│ └── wwwroot/ # filled at build: SPA dist + client.tar.gz
├── frontend/ # React + Vite source
│ ├── package.json
│ └── src/
├── client/ # Python package
│ ├── requirements.txt # sounddevice, numpy, scipy, onnxruntime, websockets, ...
│ ├── requirements-nodeps.txt # openwakeword
│ ├── main.py audio.py wakeword.py session.py playback.py state.py config.py pair.py log.py
│ └── systemd/assistant.service
├── Dockerfile
└── deploy.json # Coolify config (project/server uuids, domain)
```
## Open items deferred to implementation plans
- Exact OpenAI Realtime model + voice default (`SystemSettings.DefaultModel`/`DefaultVoice`) — pick at implementation time from currently-available choices.
- Whether wake.wav / sleep.wav playback uses the persistent OutputStream (preferred — same path as session audio) or a separate one-shot — settled in the client plan.
- ASP.NET data-protection key seeding (initial bootstrap on first run).
- React UI component library — Tailwind only, or Tailwind + a small headless component lib (Radix / shadcn).
- Exact mixer control name for `set_volume` on the LISTENAI device — confirm with `amixer scontrols` on the Pi during implementation.
## Reference paths
- Prototype spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`
- Prototype findings: `findings.md`
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
- Coolify skills: `deploying-to-coolify-via-api`, `preparing-dotnet-react-app-for-coolify`
Reference in New Issue View Git Blame Copy Permalink