Smart-assistant main-build design spec
Captures the architecture for the post-prototype build: Pi Python client + ASP.NET backend (with embedded React UI) on Coolify, SQLite, multi-turn Realtime sessions with end_session/get_current_time/set_volume tools, pairing-code device onboarding, and idempotent install/update flow.
This commit is contained in:
@@ -0,0 +1,516 @@
|
||||
# Smart Assistant — design
|
||||
|
||||
**Date:** 2026-06-11
|
||||
**Status:** approved (brainstorming complete)
|
||||
**Scope:** main build of the voice assistant, replacing the throwaway prototypes in `tests/`. Two deliverables: a Python client on the Raspberry Pi, and an ASP.NET backend (with embedded React management UI) deployed to Coolify. Brings up everything needed to wake on a wakeword, hold a multi-turn conversation through the OpenAI Realtime API, persist a text history per device, run a small built-in tool registry, and let users manage their devices through a browser.
|
||||
|
||||
## Context
|
||||
|
||||
The prototype phase (`docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`, results in `findings.md`) proved that recording/playback and openwakeword both work on the target Pi. The main assistant is built on top of those findings.
|
||||
|
||||
Key carry-forwards from the prototypes:
|
||||
|
||||
- USB Speaker Phone (ALSA card 3, Anhui LISTENAI) is the only mic/speaker. 48 kHz native; `PA_ALSA_PLUGHW=1` set before importing `sounddevice` lets us open at any rate.
|
||||
- `python-sounddevice` + `numpy` are the audio stack.
|
||||
- `openwakeword` runs at ~0.6 s load, well under 50% of one core; the `alexa` stock model is reused for v1 (custom-trained wakeword is a separate follow-up task).
|
||||
- `openwakeword==0.6.0` needs `--no-deps` install on Python 3.13 because its `tflite-runtime` pin has no wheel; the prototype's two-pass install pattern is kept.
|
||||
- Speaker hears its own output → detector must be gated while playing audio, and uplink to OpenAI must be muted while the assistant is speaking.
|
||||
- TTS/beep must not block the input stream — callback-driven InputStream + persistent OutputStream is the pattern.
|
||||
|
||||
## Goals
|
||||
|
||||
1. A fresh Raspberry Pi can be paired to a user account by running one curl-piped install command and entering a short code.
|
||||
2. The Pi wakes on the wakeword, opens a Realtime conversation, supports multi-turn until the user says "bye" or goes idle, and plays optional `wake.wav` / `sleep.wav` cues.
|
||||
3. A signed-in user sees only their own devices and conversations in the management UI.
|
||||
4. An admin user can manage system-wide defaults and other users.
|
||||
5. Per-device configuration: name, system prompt override, voice, model, idle timeout, enabled-tools toggles. Changes take effect on the next session.
|
||||
6. Text-only conversation history per device, viewable in the UI.
|
||||
7. Built-in C# tool registry with `end_session`, `get_current_time`, `set_volume` shipped in v1.
|
||||
8. Backend deployed to Coolify as one container with one domain; deploys triggered fire-and-forget via the Coolify v4 API.
|
||||
9. The install script is the update mechanism: re-running it upgrades the client in place without re-pairing.
|
||||
|
||||
## Non-goals (v1)
|
||||
|
||||
- Long-term memory across sessions.
|
||||
- MCP-based external tools (only built-in C# tools).
|
||||
- Audio recordings of conversations (text only).
|
||||
- Live conversation tailing or audio meters in the UI.
|
||||
- Per-device upload of `wake.wav`/`sleep.wav` via UI (filesystem only — v1.1).
|
||||
- Custom-trained wakeword (uses the stock `alexa` model for v1).
|
||||
- Barge-in (interrupting the assistant mid-reply).
|
||||
- Multi-language (English only).
|
||||
- Offline / local-only fallback when the network is down.
|
||||
- OIDC sign-in (local ASP.NET Identity only).
|
||||
- `.deb` packaging for the client.
|
||||
- Load / performance testing.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Topology
|
||||
|
||||
```
|
||||
┌─────────────────────┐ pair code / token ┌──────────────────────────────────────────┐
|
||||
│ Browser (admin UI) │ ──────HTTPS (REST)────────▶ │ │
|
||||
│ React SPA │ │ ASP.NET backend (Coolify container) │
|
||||
└─────────────────────┘ │ │
|
||||
│ ┌────────────┐ ┌────────────────────┐ │
|
||||
┌─────────────────────┐ WSS audio + control │ │ REST/UI │ │ Device hub │ │
|
||||
│ Pi client (Python) │ ◀───────────────────────────▶│ │ + Identity│ │ (per-device WS) │ │
|
||||
│ wakeword + audio │ │ └────────────┘ └─────────┬──────────┘ │
|
||||
└─────────────────────┘ │ │ │
|
||||
│ ┌────────────┐ ┌────────▼──────────┐ │
|
||||
│ │ Tools │ │ Realtime relay │ │
|
||||
│ │ registry │◀─┤ (per-session WS │ │
|
||||
│ └────────────┘ │ upstream) │ │
|
||||
│ └─────────┬──────────┘ │
|
||||
│ SQLite (named volume) │ │
|
||||
└────────────────────────────┼────────────┘
|
||||
│ WSS
|
||||
▼
|
||||
OpenAI Realtime API
|
||||
```
|
||||
|
||||
One container; React SPA served by the same ASP.NET process. SQLite file lives on a Coolify named volume. The OpenAI API key is held server-side; clients never see it.
|
||||
|
||||
### Why backend relay (not direct Pi ↔ OpenAI)
|
||||
|
||||
- One API key, server-side only.
|
||||
- All user-side and assistant-side transcripts land in our database directly off the upstream WS events (no separate STT pass).
|
||||
- Tool definitions and changes don't require a client redeploy.
|
||||
- Backend can push audio to a Pi on its own (future feature) over the same WS.
|
||||
|
||||
### Backend
|
||||
|
||||
Single ASP.NET 9 project (`backend/`) organised into modules:
|
||||
|
||||
```
|
||||
backend/
|
||||
├── Program.cs # composition root, EF migrate, route + hub mapping
|
||||
├── Auth/ # ASP.NET Identity, roles (Admin, User), invite flow
|
||||
├── Devices/ # device CRUD, pairing-code issuance, token mgmt
|
||||
├── DeviceHub/ # WebSocket handler at /device, per-device session state
|
||||
├── Realtime/ # RealtimeRelay: per-conversation upstream WS to OpenAI
|
||||
├── Tools/ # ITool interface, registry, built-in tools
|
||||
├── Conversations/ # persistence of turns, transcripts, tool calls
|
||||
├── Install/ # serves install.sh and client.tar.gz
|
||||
├── Data/ # EF Core DbContext, migrations, SQLite
|
||||
└── wwwroot/ # built React SPA (Vite output) + client.tar.gz
|
||||
```
|
||||
|
||||
Key services (DI):
|
||||
|
||||
- `DeviceRegistry` (singleton) — in-memory map `deviceId → ActiveDevice`, tracks online state and last frame timestamp.
|
||||
- `RealtimeSessionFactory` (singleton) — given a `DeviceContext`, opens an upstream WSS to OpenAI and returns a `RealtimeSession` instance bound to one conversation.
|
||||
- `ToolRegistry` (singleton) — static list of `ITool`s registered at startup, filterable by per-device enabled set.
|
||||
- `ConversationLog` (scoped) — append-only writer for turns; flushes per event so a crash mid-conversation still preserves what was said.
|
||||
- `OpenAIKeyProvider` (singleton) — reads `OPENAI_API_KEY` from env.
|
||||
|
||||
### Pi client
|
||||
|
||||
Python package (`client/`), deployed to `~/assistant/code/` on the Pi:
|
||||
|
||||
```
|
||||
client/
|
||||
├── main.py # entry: sets PA_ALSA_PLUGHW BEFORE importing sounddevice
|
||||
├── audio.py # USB device discovery, persistent in/out streams
|
||||
├── wakeword.py # openwakeword wrapper + gating flag
|
||||
├── session.py # backend WS client: binary audio + JSON control
|
||||
├── playback.py # queue-fed playback, wake/sleep wav hooks
|
||||
├── state.py # state machine
|
||||
├── config.py # loads ~/assistant/state/config.json
|
||||
├── pair.py # one-shot: prompt for code, hit /api/pair, write config.json
|
||||
└── log.py # structured logging to journald
|
||||
```
|
||||
|
||||
State persists at `~/assistant/state/`:
|
||||
- `config.json` (mode 0600) — `device_id`, `device_token`, `backend_ws`
|
||||
- `wake.wav` / `sleep.wav` — optional cues, played if present
|
||||
- (room for future per-device assets; never rsynced/overwritten by deploys)
|
||||
|
||||
`~/assistant/code/` is the only directory that the install script replaces on update; `~/assistant/state/` is untouched.
|
||||
|
||||
### State machine
|
||||
|
||||
```
|
||||
wakeword fires session_started
|
||||
┌─────────────────────┐ ┌──────────────────────┐
|
||||
│ ▼ │ ▼
|
||||
[IDLE] ─────────► [WAKE_PENDING] ─────► [LISTENING] ───────► [ASSISTANT_SPEAKING]
|
||||
▲ │ ▲ │
|
||||
│ │ │ assistant done │
|
||||
│ └───┴──────────────────┘
|
||||
│ session_ended
|
||||
└─────────────────────────────── (idle timeout OR end_session tool)
|
||||
plays sleep.wav, re-enables wakeword
|
||||
```
|
||||
|
||||
- **IDLE** — wakeword detector enabled, no upstream audio. Persistent backend WS open with heartbeats.
|
||||
- **WAKE_PENDING** — wakeword fired; play `wake.wav` (non-blocking); send `wake`; wait for `session_started`. Wakeword disabled.
|
||||
- **LISTENING** — mic frames stream to backend; server-VAD on OpenAI decides turn boundaries; we don't compute local VAD.
|
||||
- **ASSISTANT_SPEAKING** — uplink muted; inbound frames flow to playback queue.
|
||||
- On `assistant_done` from backend → LISTENING again (uplink unmuted).
|
||||
- On `session_ended` from backend → wait for current audio to drain, play `sleep.wav`, send `playback_done(for=sleep)` → IDLE.
|
||||
|
||||
### Audio plumbing
|
||||
|
||||
- **Wire format**: PCM16 little-endian, 24 kHz mono, exactly 1920 samples (3840 bytes, 80 ms) per binary frame. Matches OpenAI Realtime's default `pcm16`. No transcoding in the backend.
|
||||
- **Capture**: single InputStream, callback-driven, blocksize 1920, sample rate 24 000 Hz. Callback only enqueues — never blocks.
|
||||
- **Playback**: single OutputStream opened at startup, persistent. Playback worker pulls from an `asyncio.Queue` and writes — never `sd.play()/sd.wait()` (which would block the InputStream the way the prototype did).
|
||||
- **Wakeword feed**: each captured 1920-sample frame is downsampled to 1280 samples (80 ms at 16 kHz) via `scipy.signal.resample_poly(frame, up=2, down=3)` and fed to `model.predict()`.
|
||||
- `PA_ALSA_PLUGHW=1` is set at the very top of `main.py` before any `sounddevice` import.
|
||||
|
||||
### Threading model
|
||||
|
||||
- Main thread runs an asyncio loop owning the backend WS.
|
||||
- PortAudio invokes the InputStream callback on its own thread; the callback writes mic frames to a bounded `queue.Queue`. On overflow, oldest frame is dropped and a warning is logged.
|
||||
- A bridge thread reads `queue.Queue`, runs wakeword.predict() when state is IDLE/WAKE_PENDING, and forwards bytes to the asyncio loop via `loop.call_soon_threadsafe` for upstream sends when state is LISTENING.
|
||||
- A playback worker (asyncio task) drains an `asyncio.Queue` and writes to the OutputStream in a thread executor.
|
||||
|
||||
### Reconnect & heartbeat
|
||||
|
||||
- Pi → backend WS reconnects with exponential backoff 1 → 30 s.
|
||||
- While disconnected, state stays IDLE; wakewords are ignored.
|
||||
- An in-flight conversation does **not** survive a network drop — backend marks conversation `EndReason="error"`.
|
||||
- Client sends `{"type":"ping"}` every 15 s when IDLE; backend uses it to update `Devices.LastSeenAt` and drive the UI online indicator.
|
||||
|
||||
## Wire protocol (Pi ↔ backend)
|
||||
|
||||
One persistent WebSocket at `wss://<backend>/device`. Authenticated via `Authorization: Bearer <device-token>` on the upgrade request; connection rejected if the token is revoked or unknown.
|
||||
|
||||
### Binary frames
|
||||
|
||||
Raw PCM16 LE, 24 kHz mono, exactly 1920 samples (3840 bytes). No header. Direction inferred from sender. Backend drops binary frames received outside an active session. Wrong-sized frames are dropped silently with a log entry (protects against runaways without surfacing an error).
|
||||
|
||||
### Text frames (JSON envelopes)
|
||||
|
||||
Client → backend:
|
||||
|
||||
| `type` | Fields | Notes |
|
||||
| ----------------- | ----------------------------------- | ----------------------------------------------------------- |
|
||||
| `hello` | `client_version`, `device_id` | Sent on connect; backend replies `hello_ack` with the device's config snapshot. |
|
||||
| `ping` | — | Every 15 s while IDLE. Backend replies `pong`. |
|
||||
| `wake` | `at` (client ms timestamp) | Triggers backend to open upstream Realtime WS. |
|
||||
| `cancel` | — | User-initiated abort (e.g. SIGINT). Backend closes session. |
|
||||
| `playback_done` | `for` (`wake` \| `sleep` \| `assistant_turn`) | Lets backend know when audio finished playing on the Pi (needed to time `session_ended` after `sleep.wav`). |
|
||||
| `tool_result` | `call_id`, `ok`, `result?`, `error?` | Reply to a Pi-side tool call (e.g. `set_volume`). |
|
||||
|
||||
Backend → client:
|
||||
|
||||
| `type` | Fields | Notes |
|
||||
| ------------------ | ----------------------------------- | ---------------------------------------------------------- |
|
||||
| `hello_ack` | `config` (voice/model/prompt/tools/idle_timeout) | Client uses this to set local state. |
|
||||
| `pong` | — | |
|
||||
| `config_updated` | `config` | Pushed when admin saves changes in the UI; takes effect next session. |
|
||||
| `session_started` | `conversation_id` | Realtime upstream is ready; client may start uplink. |
|
||||
| `assistant_done` | — | Assistant turn ended; client unmutes uplink. |
|
||||
| `session_ended` | `reason` (`tool` \| `idle` \| `error`) | Client plays `sleep.wav` (if present), then `playback_done(for=sleep)`. |
|
||||
| `tool_call` | `call_id`, `name`, `arguments` | Invoked for tools whose execution requires the Pi. |
|
||||
| `play_audio` | `kind` (`session` \| `notification`) | Marker; subsequent binary frames are payload. Ended by `play_audio_end`. |
|
||||
| `play_audio_end` | — | End-of-payload marker for `play_audio(kind=notification)`. |
|
||||
| `error` | `code`, `message`, `fatal?` | Logged on client; non-fatal unless `fatal=true`. |
|
||||
|
||||
Uplink muting is purely a client-side decision based on state. Backend doesn't enforce it.
|
||||
|
||||
## Conversation lifecycle
|
||||
|
||||
```
|
||||
PI BACKEND OPENAI REALTIME
|
||||
─────────────────────────────────────────────────────────────────────────────────
|
||||
state=IDLE
|
||||
mic frames → wakeword.predict
|
||||
wakeword fires
|
||||
play wake.wav (non-blocking)
|
||||
send {type:"wake"} ──►
|
||||
open WSS to OpenAI ──►
|
||||
send session.update ──►
|
||||
(voice, instructions,
|
||||
input_audio_transcription,
|
||||
tools=[end_session,
|
||||
get_current_time,
|
||||
set_volume])
|
||||
◄── session.updated
|
||||
insert Conversations row
|
||||
send {type:"session_started",
|
||||
conversation_id} ──►
|
||||
state=LISTENING
|
||||
mic frames (binary) ──────────────► forward as input_audio_buffer.append ─►
|
||||
◄── input_audio_buffer.speech_started
|
||||
◄── input_audio_buffer.speech_stopped
|
||||
◄── response.created
|
||||
◄── response.audio.delta (x N)
|
||||
binary frames ◄─────────────── forward to PI as binary
|
||||
binary frames ──► playback queue
|
||||
state=ASSISTANT_SPEAKING
|
||||
(uplink muted)
|
||||
◄── response.audio_transcript.done
|
||||
(persist assistant Turn)
|
||||
◄── response.done
|
||||
send {type:"assistant_done"} ──►
|
||||
state=LISTENING
|
||||
◄── conversation.item.input_audio_transcription.completed
|
||||
(persist user Turn)
|
||||
...follow-up turn(s)...
|
||||
|
||||
Path A — model calls end_session
|
||||
◄── response.function_call_arguments.done
|
||||
(name="end_session")
|
||||
◄── response.audio.delta (the "bye" reply)
|
||||
forward audio to PI ──────────►
|
||||
wait for response.done
|
||||
◄── response.done
|
||||
send {type:"assistant_done"} ──►
|
||||
send {type:"session_ended",
|
||||
reason="tool"} ──►
|
||||
close upstream WSS
|
||||
persist Conversations.EndReason
|
||||
state=IDLE-PENDING
|
||||
wait for current audio buffer drain
|
||||
play sleep.wav
|
||||
send {type:"playback_done",
|
||||
for:"sleep"} ──►
|
||||
state=IDLE
|
||||
(wakeword re-enabled)
|
||||
|
||||
Path B — idle timeout
|
||||
no mic activity for IdleTimeoutSeconds while LISTENING
|
||||
send {type:"session_ended",
|
||||
reason="idle"} ──►
|
||||
close upstream WSS
|
||||
(same sleep.wav + transition as Path A)
|
||||
```
|
||||
|
||||
### `end_session` sequencing
|
||||
|
||||
The tool call arrives *during* the assistant's response. The relay must:
|
||||
|
||||
1. Forward all audio deltas until `response.done`.
|
||||
2. Then send `assistant_done`, then `session_ended(reason=tool)`.
|
||||
3. NOT execute the tool mid-response — `end_session` has no real side effect; the relay records "the model asked to end" and closes after the audio finishes. The tool result returned to OpenAI is `{"ok": true}` (for cleanliness; the upstream is about to close).
|
||||
|
||||
## Tools registry
|
||||
|
||||
C# interface, registered as singletons at startup:
|
||||
|
||||
```csharp
|
||||
public interface ITool
|
||||
{
|
||||
string Name { get; } // OpenAI tool name
|
||||
string Description { get; } // shown to the model
|
||||
JsonSchema ParameterSchema { get; } // OpenAI-compatible
|
||||
Task<ToolResult> ExecuteAsync(
|
||||
ToolInvocation invocation,
|
||||
DeviceContext device,
|
||||
CancellationToken ct);
|
||||
}
|
||||
|
||||
public record ToolInvocation(string CallId, JsonElement Arguments);
|
||||
public record ToolResult(JsonElement Output); // Output is JSON sent back to model
|
||||
public record DeviceContext(Guid DeviceId, Guid ConversationId, IDeviceChannel Channel);
|
||||
```
|
||||
|
||||
`IDeviceChannel` lets a tool send `tool_call`-style messages to the Pi and await `tool_result`. Tools that don't need the Pi simply don't use it.
|
||||
|
||||
### v1 built-in tools
|
||||
|
||||
1. **`end_session`** — no parameters. Sets a "model wants to end" flag on the active `RealtimeSession`; the relay's `response.done` handler closes the upstream WS and emits `session_ended(reason=tool)`. Returns `{"ok": true}`.
|
||||
|
||||
2. **`get_current_time`** — no parameters. Returns `{"now": "<ISO-8601 UTC>"}` from `DateTimeOffset.UtcNow`. Smoke test for the tool plane.
|
||||
|
||||
3. **`set_volume`** — `level: integer 0..100`. Backend sends `tool_call(name=set_volume, arguments={level})` to the Pi; Pi runs `amixer set 'PCM' N%` (or device-specific mixer) and replies with `tool_result(call_id, ok, error?)`. Backend forwards back to OpenAI as the function call output.
|
||||
|
||||
The per-device enabled set is enforced by filtering the tool list in `session.update`. Disabled tools are simply not advertised to the model.
|
||||
|
||||
## Data model (SQLite via EF Core)
|
||||
|
||||
```
|
||||
AspNetUsers, AspNetRoles, ... # Identity standard
|
||||
Devices
|
||||
Id (Guid) OwnerUserId Name PairedAt LastSeenAt TokenHash IsRevoked
|
||||
DeviceConfigs # 1:1 with Devices
|
||||
DeviceId SystemPrompt Voice Model IdleTimeoutSeconds EnabledToolsJson
|
||||
PairingCodes
|
||||
Code (8 chars) OwnerUserId ExpiresAt ConsumedAt
|
||||
Conversations
|
||||
Id DeviceId StartedAt EndedAt EndReason ("tool" | "idle" | "error")
|
||||
Turns
|
||||
Id ConversationId Index Role ("user" | "assistant" | "tool") Text ToolName ToolArgsJson ToolResultJson CreatedAt
|
||||
SystemSettings # singleton row
|
||||
DefaultVoice DefaultModel DefaultSystemPrompt
|
||||
```
|
||||
|
||||
- `TokenHash` = SHA-256 of the device token; plaintext is shown once during pairing and never persisted.
|
||||
- `EnabledToolsJson` is a small JSON array of tool names; no relational table needed.
|
||||
- `Turns.Index` is monotonic per-conversation for ordering.
|
||||
- `Database.Migrate()` runs on startup; SQLite file at `/data/assistant.db` on a Coolify named volume.
|
||||
|
||||
## Pairing flow
|
||||
|
||||
UI side:
|
||||
|
||||
1. User signs in, clicks **Add device**, enters a friendly name.
|
||||
2. Backend generates an 8-char alphanumeric code from a 32-char alphabet (no `0/O/1/l/I`); stores `PairingCodes(Code, OwnerUserId, ExpiresAt = NOW + 10 min)`.
|
||||
3. UI shows the code and the one-line installer:
|
||||
```
|
||||
curl -fsSL https://<backend>/install.sh | bash
|
||||
```
|
||||
4. Code stays visible on the device page until consumed or expired; regenerate at any time.
|
||||
|
||||
Pi side, first run:
|
||||
|
||||
1. Installer runs.
|
||||
2. If `~/assistant/state/config.json` exists with a valid `device_token`, **skip pairing** and proceed to update path.
|
||||
3. Otherwise prompt `Pairing code: `.
|
||||
4. `POST https://<backend>/api/pair` with `{ "code", "hostname", "client_version" }`.
|
||||
5. Backend validates code (exists, unexpired, unconsumed), creates `Devices(OwnerUserId = code.OwnerUserId, Name = <user-set-name>, TokenHash = sha256(newToken))`, marks code `ConsumedAt = NOW`, creates default `DeviceConfigs` row from `SystemSettings`, returns `{ "device_id", "device_token", "backend_ws" }`.
|
||||
6. Installer writes `~/assistant/state/config.json` (mode 0600).
|
||||
|
||||
Security:
|
||||
- Pairing codes are one-shot, 10-min TTL, 8 chars from a 32-char alphabet (~40 bits — fine for the TTL window).
|
||||
- Device tokens are 32 bytes random, base64url-encoded, stored only as SHA-256 hash server-side.
|
||||
|
||||
## Install script
|
||||
|
||||
`install.sh` is served by the backend at `GET /install.sh` and is fully idempotent.
|
||||
|
||||
```
|
||||
1. Resolve BACKEND_URL from the script's own served-from origin (no hardcoding).
|
||||
2. apt-get install -y python3-venv libportaudio2 alsa-utils
|
||||
(sudo prompt OK; passwordless-sudo per findings.md §E is the recommended setup).
|
||||
3. mkdir -p ~/assistant/{code,state}
|
||||
4. python3 -m venv ~/assistant/.venv (if missing)
|
||||
5. curl -fsSL $BACKEND_URL/client.tar.gz | tar -xz -C ~/assistant/code
|
||||
6. ~/assistant/.venv/bin/pip install -r ~/assistant/code/requirements.txt
|
||||
7. ~/assistant/.venv/bin/pip install --no-deps -r ~/assistant/code/requirements-nodeps.txt
|
||||
(openwakeword install split per findings.md §3).
|
||||
8. If config.json missing → run `~/assistant/.venv/bin/python -m client.pair`,
|
||||
which prompts for the code and writes config.json.
|
||||
9. Install systemd user unit at ~/.config/systemd/user/assistant.service.
|
||||
10. systemctl --user daemon-reload
|
||||
11. systemctl --user enable --now assistant
|
||||
12. loginctl enable-linger pi (so the unit runs without an active login).
|
||||
13. Print status; if pairing happened, print "Paired to <backend> as device <name>".
|
||||
```
|
||||
|
||||
The systemd user unit runs `~/assistant/.venv/bin/python -m client.main` with:
|
||||
- `Restart=on-failure`
|
||||
- `RestartSec=5`
|
||||
- `Environment=PA_ALSA_PLUGHW=1`
|
||||
|
||||
### Updates
|
||||
|
||||
Re-running the same one-liner upgrades the client. Steps 5–7 fetch and reinstall the latest bundle; step 11 restarts the unit. Pairing is skipped because `config.json` already exists.
|
||||
|
||||
`curl ... | bash -s -- --reset` deletes `config.json` and forces a fresh pairing.
|
||||
|
||||
### Backend install endpoints
|
||||
|
||||
- `GET /install.sh` — returns the shell script with `BACKEND_URL` baked from the request's `X-Forwarded-Host` / `Host`. Public, no auth.
|
||||
- `GET /client.tar.gz` — returns the current client bundle (built at deploy time into `wwwroot/`). Public, no auth — the code is not a secret; the device token gates the device WS.
|
||||
- `POST /api/pair` — accepts pairing code, returns device token. No prior auth (code is the credential).
|
||||
|
||||
The bundle is built during the Docker image build: a stage runs `tar czf client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt` and copies the result into `wwwroot/`.
|
||||
|
||||
## Error handling
|
||||
|
||||
| Scenario | Detection | Backend action | Client action |
|
||||
| --- | --- | --- | --- |
|
||||
| Pi loses network | WS close / heartbeat miss | Close upstream Realtime WS if active, log conversation `EndReason="error"`, mark device offline | Exponential backoff reconnect 1→30 s, stay IDLE, no wakeword |
|
||||
| Backend restarts | Pi WS closes from server side | (n/a — process gone) | Same backoff reconnect; on reconnect send fresh `hello` |
|
||||
| OpenAI WS drops mid-conversation | upstream `on_close` | Send `error(code=upstream)` + `session_ended(reason=error)`, persist partial transcript | Play `sleep.wav`, return to IDLE |
|
||||
| OpenAI rate limit / 429 on connect | upstream open fails | Don't open session; send `error(code=upstream, message=...)` instead of `session_started` | Play `sleep.wav`, IDLE |
|
||||
| Tool execution exception (server-side) | C# try/catch around `ITool.ExecuteAsync` | Send tool result `{"error": "..."}` back to OpenAI so the model can recover; log | (no client involvement) |
|
||||
| Pi-side tool fails (e.g. `set_volume` amixer error) | `tool_result.ok=false` | Forward `{"error": "..."}` to OpenAI | Log and continue |
|
||||
| Device token revoked while connected | Admin clicks revoke | Set `IsRevoked=true`, close all WS for that device with `error(code=auth_revoked, fatal=true)` | Stop the process (systemd will restart, fail to auth, sleep 60 s, retry — visible in logs) |
|
||||
| SQLite write failure | EF exception | Bubble up to REST as 500; on the hub side, drop the offending log event, keep session running | (n/a) |
|
||||
| Two simultaneous wakeword fires | `wake` while not IDLE | Backend ignores second `wake` (idempotent within active session) | Client also gates locally |
|
||||
| Audio buffer overflow on Pi | PortAudio callback queue full | (n/a) | Drop oldest frame, log a warning |
|
||||
|
||||
## Management UI (v1)
|
||||
|
||||
React SPA, served from the backend's `wwwroot/`. Pages:
|
||||
|
||||
- **Sign in / sign up** (ASP.NET Identity).
|
||||
- **Devices** — list of the signed-in user's devices: name, online indicator, last-seen.
|
||||
- **Device detail** — rename; generate pairing code (with countdown); revoke; per-device config form: system prompt override, voice, model, idle timeout slider, enabled-tools checkboxes; conversation history viewer (paginated, text only).
|
||||
- **Admin** (admin role only) — user list, invite/disable users, global model/voice/prompt defaults.
|
||||
|
||||
Out of v1: live conversation tail, audio meters, audio recordings, memory UI, per-device `wake.wav`/`sleep.wav` upload (filesystem only for v1; v1.1 adds upload).
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- **Backend unit tests** (xUnit): `ToolRegistry` filtering by enabled set; `PairingService` (code generation, expiry, single-use); `ConversationLog` (turn ordering, persistence); auth/role enforcement on the admin endpoints.
|
||||
- **Backend integration tests** (`WebApplicationFactory`): full pair flow end-to-end against in-memory SQLite; device-hub WS with a fake OpenAI upstream (test double that emits scripted Realtime events).
|
||||
- **Client unit tests** (pytest): state machine transitions (pure-Python, no audio); resample correctness against golden samples; config loading. Audio I/O and wakeword are not unit-tested — already covered by the prototypes and by manual end-to-end.
|
||||
- **End-to-end manual smoke test** (in repo `README.md`): pair a fresh Pi, say wakeword, ask "what time is it", say "bye", verify conversation row + sleep.wav heard. Run after any change that touches the conversation flow.
|
||||
- No load / perf testing.
|
||||
|
||||
## Deployment (Coolify)
|
||||
|
||||
Single Docker image, multi-stage Dockerfile (`preparing-dotnet-react-app-for-coolify` skill is the reference for the layout):
|
||||
|
||||
1. **Frontend build** stage — Node, `npm ci && npm run build` against `frontend/`.
|
||||
2. **Backend build** stage — .NET SDK, `dotnet publish backend/ -c Release -o /publish`.
|
||||
3. **Client bundle** stage — `tar czf /publish/wwwroot/client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt`.
|
||||
4. **Final** stage — `mcr.microsoft.com/dotnet/aspnet:9.0`, copies `/publish`, copies SPA `dist/` into `wwwroot/`, sets `ASPNETCORE_URLS=http://+:8080`, exposes 8080.
|
||||
|
||||
Coolify app (provisioned via `deploying-to-coolify-via-api`):
|
||||
|
||||
- Source: this repo on Gitea.
|
||||
- Expose 8080 (Traefik fronts it).
|
||||
- `SERVICE_FQDN_BACKEND` → backend domain.
|
||||
- Env vars:
|
||||
- `OPENAI_API_KEY` — secret.
|
||||
- `ConnectionStrings__Default=Data Source=/data/assistant.db`
|
||||
- `ASPNETCORE_URLS=http://+:8080`
|
||||
- Named volumes:
|
||||
- `/data` — SQLite file.
|
||||
- `/keys` — ASP.NET data-protection keys (cookies survive container restarts).
|
||||
- Deploys triggered fire-and-forget via the Coolify v4 API (per global CLAUDE.md rule).
|
||||
|
||||
## Repo layout (final)
|
||||
|
||||
```
|
||||
Assistant/
|
||||
├── CLAUDE.md
|
||||
├── README.md # smoke-test instructions
|
||||
├── findings.md # prototype-phase findings (kept)
|
||||
├── tests/ # prototype throwaways (kept for reference)
|
||||
│ ├── 01-record-play/
|
||||
│ └── 02-wakeword/
|
||||
├── docs/superpowers/
|
||||
│ ├── specs/
|
||||
│ │ ├── 2026-06-11-voice-assistant-prototypes-design.md
|
||||
│ │ └── 2026-06-11-smart-assistant-design.md # this design
|
||||
│ └── plans/
|
||||
│ └── (one or more implementation plans, written next)
|
||||
├── backend/ # ASP.NET solution
|
||||
│ ├── backend.csproj
|
||||
│ ├── Program.cs
|
||||
│ ├── Auth/ Devices/ DeviceHub/ Realtime/ Tools/ Conversations/ Install/ Data/
|
||||
│ └── wwwroot/ # filled at build: SPA dist + client.tar.gz
|
||||
├── frontend/ # React + Vite source
|
||||
│ ├── package.json
|
||||
│ └── src/
|
||||
├── client/ # Python package
|
||||
│ ├── requirements.txt # sounddevice, numpy, scipy, onnxruntime, websockets, ...
|
||||
│ ├── requirements-nodeps.txt # openwakeword
|
||||
│ ├── main.py audio.py wakeword.py session.py playback.py state.py config.py pair.py log.py
|
||||
│ └── systemd/assistant.service
|
||||
├── Dockerfile
|
||||
└── deploy.json # Coolify config (project/server uuids, domain)
|
||||
```
|
||||
|
||||
## Open items deferred to implementation plans
|
||||
|
||||
- Exact OpenAI Realtime model + voice default (`SystemSettings.DefaultModel`/`DefaultVoice`) — pick at implementation time from currently-available choices.
|
||||
- Whether wake.wav / sleep.wav playback uses the persistent OutputStream (preferred — same path as session audio) or a separate one-shot — settled in the client plan.
|
||||
- ASP.NET data-protection key seeding (initial bootstrap on first run).
|
||||
- React UI component library — Tailwind only, or Tailwind + a small headless component lib (Radix / shadcn).
|
||||
- Exact mixer control name for `set_volume` on the LISTENAI device — confirm with `amixer scontrols` on the Pi during implementation.
|
||||
|
||||
## Reference paths
|
||||
|
||||
- Prototype spec: `docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md`
|
||||
- Prototype findings: `findings.md`
|
||||
- Pi access: `pi@192.168.50.115`, password `assistant` (see `CLAUDE.md`)
|
||||
- Coolify skills: `deploying-to-coolify-via-api`, `preparing-dotnet-react-app-for-coolify`
|
||||
Reference in New Issue
Block a user