tes/Assistant
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
55257b8867f980417e23cd0b213c073af39db62d
Assistant/docs/superpowers/specs/2026-06-11-smart-assistant-design.md
T
tes 99a9950cee Smart-assistant main-build design spec 
Captures the architecture for the post-prototype build: Pi Python client +
ASP.NET backend (with embedded React UI) on Coolify, SQLite, multi-turn
Realtime sessions with end_session/get_current_time/set_volume tools,
pairing-code device onboarding, and idempotent install/update flow.
2026-06-11 15:48:10 +00:00

33 KiB
Raw Blame History

Smart Assistant — design

Date: 2026-06-11 Status: approved (brainstorming complete) Scope: main build of the voice assistant, replacing the throwaway prototypes in tests/. Two deliverables: a Python client on the Raspberry Pi, and an ASP.NET backend (with embedded React management UI) deployed to Coolify. Brings up everything needed to wake on a wakeword, hold a multi-turn conversation through the OpenAI Realtime API, persist a text history per device, run a small built-in tool registry, and let users manage their devices through a browser.

Context

The prototype phase (docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md, results in findings.md) proved that recording/playback and openwakeword both work on the target Pi. The main assistant is built on top of those findings.

Key carry-forwards from the prototypes:

  • USB Speaker Phone (ALSA card 3, Anhui LISTENAI) is the only mic/speaker. 48 kHz native; PA_ALSA_PLUGHW=1 set before importing sounddevice lets us open at any rate.
  • python-sounddevice + numpy are the audio stack.
  • openwakeword runs at ~0.6 s load, well under 50% of one core; the alexa stock model is reused for v1 (custom-trained wakeword is a separate follow-up task).
  • openwakeword==0.6.0 needs --no-deps install on Python 3.13 because its tflite-runtime pin has no wheel; the prototype's two-pass install pattern is kept.
  • Speaker hears its own output → detector must be gated while playing audio, and uplink to OpenAI must be muted while the assistant is speaking.
  • TTS/beep must not block the input stream — callback-driven InputStream + persistent OutputStream is the pattern.

Goals

  1. A fresh Raspberry Pi can be paired to a user account by running one curl-piped install command and entering a short code.
  2. The Pi wakes on the wakeword, opens a Realtime conversation, supports multi-turn until the user says "bye" or goes idle, and plays optional wake.wav / sleep.wav cues.
  3. A signed-in user sees only their own devices and conversations in the management UI.
  4. An admin user can manage system-wide defaults and other users.
  5. Per-device configuration: name, system prompt override, voice, model, idle timeout, enabled-tools toggles. Changes take effect on the next session.
  6. Text-only conversation history per device, viewable in the UI.
  7. Built-in C# tool registry with end_session, get_current_time, set_volume shipped in v1.
  8. Backend deployed to Coolify as one container with one domain; deploys triggered fire-and-forget via the Coolify v4 API.
  9. The install script is the update mechanism: re-running it upgrades the client in place without re-pairing.

Non-goals (v1)

  • Long-term memory across sessions.
  • MCP-based external tools (only built-in C# tools).
  • Audio recordings of conversations (text only).
  • Live conversation tailing or audio meters in the UI.
  • Per-device upload of wake.wav/sleep.wav via UI (filesystem only — v1.1).
  • Custom-trained wakeword (uses the stock alexa model for v1).
  • Barge-in (interrupting the assistant mid-reply).
  • Multi-language (English only).
  • Offline / local-only fallback when the network is down.
  • OIDC sign-in (local ASP.NET Identity only).
  • .deb packaging for the client.
  • Load / performance testing.

Architecture

Topology

┌─────────────────────┐     pair code / token       ┌──────────────────────────────────────────┐
│ Browser (admin UI)  │ ──────HTTPS (REST)────────▶ │                                          │
│  React SPA          │                              │     ASP.NET backend (Coolify container)  │
└─────────────────────┘                              │                                          │
                                                      │  ┌────────────┐  ┌────────────────────┐ │
┌─────────────────────┐     WSS audio + control      │  │  REST/UI   │  │  Device hub        │ │
│ Pi client (Python)  │ ◀───────────────────────────▶│  │  + Identity│  │  (per-device WS)   │ │
│  wakeword + audio   │                              │  └────────────┘  └─────────┬──────────┘ │
└─────────────────────┘                              │                            │            │
                                                      │  ┌────────────┐  ┌────────▼──────────┐ │
                                                      │  │  Tools     │  │  Realtime relay   │ │
                                                      │  │  registry  │◀─┤  (per-session WS  │ │
                                                      │  └────────────┘  │   upstream)       │ │
                                                      │                  └─────────┬──────────┘ │
                                                      │  SQLite (named volume)     │            │
                                                      └────────────────────────────┼────────────┘
                                                                                   │ WSS
                                                                                   ▼
                                                                          OpenAI Realtime API

One container; React SPA served by the same ASP.NET process. SQLite file lives on a Coolify named volume. The OpenAI API key is held server-side; clients never see it.

Why backend relay (not direct Pi ↔ OpenAI)

  • One API key, server-side only.
  • All user-side and assistant-side transcripts land in our database directly off the upstream WS events (no separate STT pass).
  • Tool definitions and changes don't require a client redeploy.
  • Backend can push audio to a Pi on its own (future feature) over the same WS.

Backend

Single ASP.NET 9 project (backend/) organised into modules:

backend/
├── Program.cs                       # composition root, EF migrate, route + hub mapping
├── Auth/                            # ASP.NET Identity, roles (Admin, User), invite flow
├── Devices/                         # device CRUD, pairing-code issuance, token mgmt
├── DeviceHub/                       # WebSocket handler at /device, per-device session state
├── Realtime/                        # RealtimeRelay: per-conversation upstream WS to OpenAI
├── Tools/                           # ITool interface, registry, built-in tools
├── Conversations/                   # persistence of turns, transcripts, tool calls
├── Install/                         # serves install.sh and client.tar.gz
├── Data/                            # EF Core DbContext, migrations, SQLite
└── wwwroot/                         # built React SPA (Vite output) + client.tar.gz

Key services (DI):

  • DeviceRegistry (singleton) — in-memory map deviceId → ActiveDevice, tracks online state and last frame timestamp.
  • RealtimeSessionFactory (singleton) — given a DeviceContext, opens an upstream WSS to OpenAI and returns a RealtimeSession instance bound to one conversation.
  • ToolRegistry (singleton) — static list of ITools registered at startup, filterable by per-device enabled set.
  • ConversationLog (scoped) — append-only writer for turns; flushes per event so a crash mid-conversation still preserves what was said.
  • OpenAIKeyProvider (singleton) — reads OPENAI_API_KEY from env.

Pi client

Python package (client/), deployed to ~/assistant/code/ on the Pi:

client/
├── main.py            # entry: sets PA_ALSA_PLUGHW BEFORE importing sounddevice
├── audio.py           # USB device discovery, persistent in/out streams
├── wakeword.py        # openwakeword wrapper + gating flag
├── session.py         # backend WS client: binary audio + JSON control
├── playback.py        # queue-fed playback, wake/sleep wav hooks
├── state.py           # state machine
├── config.py          # loads ~/assistant/state/config.json
├── pair.py            # one-shot: prompt for code, hit /api/pair, write config.json
└── log.py             # structured logging to journald

State persists at ~/assistant/state/:

  • config.json (mode 0600) — device_id, device_token, backend_ws
  • wake.wav / sleep.wav — optional cues, played if present
  • (room for future per-device assets; never rsynced/overwritten by deploys)

~/assistant/code/ is the only directory that the install script replaces on update; ~/assistant/state/ is untouched.

State machine

       wakeword fires                        session_started
   ┌─────────────────────┐               ┌──────────────────────┐
   │                     ▼               │                      ▼
[IDLE] ─────────► [WAKE_PENDING] ─────► [LISTENING] ───────► [ASSISTANT_SPEAKING]
   ▲                                       │   ▲                  │
   │                                       │   │  assistant done  │
   │                                       └───┴──────────────────┘
   │             session_ended
   └───────────────────────────────  (idle timeout OR end_session tool)
                                     plays sleep.wav, re-enables wakeword
  • IDLE — wakeword detector enabled, no upstream audio. Persistent backend WS open with heartbeats.
  • WAKE_PENDING — wakeword fired; play wake.wav (non-blocking); send wake; wait for session_started. Wakeword disabled.
  • LISTENING — mic frames stream to backend; server-VAD on OpenAI decides turn boundaries; we don't compute local VAD.
  • ASSISTANT_SPEAKING — uplink muted; inbound frames flow to playback queue.
  • On assistant_done from backend → LISTENING again (uplink unmuted).
  • On session_ended from backend → wait for current audio to drain, play sleep.wav, send playback_done(for=sleep) → IDLE.

Audio plumbing

  • Wire format: PCM16 little-endian, 24 kHz mono, exactly 1920 samples (3840 bytes, 80 ms) per binary frame. Matches OpenAI Realtime's default pcm16. No transcoding in the backend.
  • Capture: single InputStream, callback-driven, blocksize 1920, sample rate 24 000 Hz. Callback only enqueues — never blocks.
  • Playback: single OutputStream opened at startup, persistent. Playback worker pulls from an asyncio.Queue and writes — never sd.play()/sd.wait() (which would block the InputStream the way the prototype did).
  • Wakeword feed: each captured 1920-sample frame is downsampled to 1280 samples (80 ms at 16 kHz) via scipy.signal.resample_poly(frame, up=2, down=3) and fed to model.predict().
  • PA_ALSA_PLUGHW=1 is set at the very top of main.py before any sounddevice import.

Threading model

  • Main thread runs an asyncio loop owning the backend WS.
  • PortAudio invokes the InputStream callback on its own thread; the callback writes mic frames to a bounded queue.Queue. On overflow, oldest frame is dropped and a warning is logged.
  • A bridge thread reads queue.Queue, runs wakeword.predict() when state is IDLE/WAKE_PENDING, and forwards bytes to the asyncio loop via loop.call_soon_threadsafe for upstream sends when state is LISTENING.
  • A playback worker (asyncio task) drains an asyncio.Queue and writes to the OutputStream in a thread executor.

Reconnect & heartbeat

  • Pi → backend WS reconnects with exponential backoff 1 → 30 s.
  • While disconnected, state stays IDLE; wakewords are ignored.
  • An in-flight conversation does not survive a network drop — backend marks conversation EndReason="error".
  • Client sends {"type":"ping"} every 15 s when IDLE; backend uses it to update Devices.LastSeenAt and drive the UI online indicator.

Wire protocol (Pi ↔ backend)

One persistent WebSocket at wss://<backend>/device. Authenticated via Authorization: Bearer <device-token> on the upgrade request; connection rejected if the token is revoked or unknown.

Binary frames

Raw PCM16 LE, 24 kHz mono, exactly 1920 samples (3840 bytes). No header. Direction inferred from sender. Backend drops binary frames received outside an active session. Wrong-sized frames are dropped silently with a log entry (protects against runaways without surfacing an error).

Text frames (JSON envelopes)

Client → backend:

type Fields Notes
hello client_version, device_id Sent on connect; backend replies hello_ack with the device's config snapshot.
ping — Every 15 s while IDLE. Backend replies pong.
wake at (client ms timestamp) Triggers backend to open upstream Realtime WS.
cancel — User-initiated abort (e.g. SIGINT). Backend closes session.
playback_done for (wake | sleep | assistant_turn) Lets backend know when audio finished playing on the Pi (needed to time session_ended after sleep.wav).
tool_result call_id, ok, result?, error? Reply to a Pi-side tool call (e.g. set_volume).

Backend → client:

type Fields Notes
hello_ack config (voice/model/prompt/tools/idle_timeout) Client uses this to set local state.
pong —
config_updated config Pushed when admin saves changes in the UI; takes effect next session.
session_started conversation_id Realtime upstream is ready; client may start uplink.
assistant_done — Assistant turn ended; client unmutes uplink.
session_ended reason (tool | idle | error) Client plays sleep.wav (if present), then playback_done(for=sleep).
tool_call call_id, name, arguments Invoked for tools whose execution requires the Pi.
play_audio kind (session | notification) Marker; subsequent binary frames are payload. Ended by play_audio_end.
play_audio_end — End-of-payload marker for play_audio(kind=notification).
error code, message, fatal? Logged on client; non-fatal unless fatal=true.

Uplink muting is purely a client-side decision based on state. Backend doesn't enforce it.

Conversation lifecycle

PI                                BACKEND                          OPENAI REALTIME
─────────────────────────────────────────────────────────────────────────────────
state=IDLE
mic frames → wakeword.predict
wakeword fires
play wake.wav (non-blocking)
send {type:"wake"}              ──►
                                  open WSS to OpenAI            ──►
                                  send session.update            ──►
                                    (voice, instructions,
                                     input_audio_transcription,
                                     tools=[end_session,
                                            get_current_time,
                                            set_volume])
                                                                  ◄── session.updated
                                  insert Conversations row
                                  send {type:"session_started",
                                        conversation_id}        ──►
state=LISTENING
mic frames (binary) ──────────────► forward as input_audio_buffer.append ─►
                                                                  ◄── input_audio_buffer.speech_started
                                                                  ◄── input_audio_buffer.speech_stopped
                                                                  ◄── response.created
                                                                  ◄── response.audio.delta (x N)
                                  binary frames ◄───────────────  forward to PI as binary
binary frames ──► playback queue
state=ASSISTANT_SPEAKING
(uplink muted)
                                                                  ◄── response.audio_transcript.done
                                                                       (persist assistant Turn)
                                                                  ◄── response.done
                                  send {type:"assistant_done"}   ──►
state=LISTENING
                                                                  ◄── conversation.item.input_audio_transcription.completed
                                                                       (persist user Turn)
...follow-up turn(s)...

  Path A — model calls end_session
                                                                  ◄── response.function_call_arguments.done
                                                                       (name="end_session")
                                                                  ◄── response.audio.delta (the "bye" reply)
                                  forward audio to PI ──────────►
                                  wait for response.done
                                                                  ◄── response.done
                                  send {type:"assistant_done"}   ──►
                                  send {type:"session_ended",
                                        reason="tool"}            ──►
                                  close upstream WSS
                                  persist Conversations.EndReason
state=IDLE-PENDING
wait for current audio buffer drain
play sleep.wav
send {type:"playback_done",
      for:"sleep"}              ──►
state=IDLE
(wakeword re-enabled)

  Path B — idle timeout
no mic activity for IdleTimeoutSeconds while LISTENING
                                  send {type:"session_ended",
                                        reason="idle"}            ──►
                                  close upstream WSS
(same sleep.wav + transition as Path A)

end_session sequencing

The tool call arrives during the assistant's response. The relay must:

  1. Forward all audio deltas until response.done.
  2. Then send assistant_done, then session_ended(reason=tool).
  3. NOT execute the tool mid-response — end_session has no real side effect; the relay records "the model asked to end" and closes after the audio finishes. The tool result returned to OpenAI is {"ok": true} (for cleanliness; the upstream is about to close).

Tools registry

C# interface, registered as singletons at startup:

public interface ITool
{
    string Name { get; }                              // OpenAI tool name
    string Description { get; }                       // shown to the model
    JsonSchema ParameterSchema { get; }               // OpenAI-compatible
    Task<ToolResult> ExecuteAsync(
        ToolInvocation invocation,
        DeviceContext device,
        CancellationToken ct);
}

public record ToolInvocation(string CallId, JsonElement Arguments);
public record ToolResult(JsonElement Output);          // Output is JSON sent back to model
public record DeviceContext(Guid DeviceId, Guid ConversationId, IDeviceChannel Channel);

IDeviceChannel lets a tool send tool_call-style messages to the Pi and await tool_result. Tools that don't need the Pi simply don't use it.

v1 built-in tools

  1. end_session — no parameters. Sets a "model wants to end" flag on the active RealtimeSession; the relay's response.done handler closes the upstream WS and emits session_ended(reason=tool). Returns {"ok": true}.

  2. get_current_time — no parameters. Returns {"now": "<ISO-8601 UTC>"} from DateTimeOffset.UtcNow. Smoke test for the tool plane.

  3. set_volume — level: integer 0..100. Backend sends tool_call(name=set_volume, arguments={level}) to the Pi; Pi runs amixer set 'PCM' N% (or device-specific mixer) and replies with tool_result(call_id, ok, error?). Backend forwards back to OpenAI as the function call output.

The per-device enabled set is enforced by filtering the tool list in session.update. Disabled tools are simply not advertised to the model.

Data model (SQLite via EF Core)

AspNetUsers, AspNetRoles, ...            # Identity standard
Devices
  Id (Guid)  OwnerUserId  Name  PairedAt  LastSeenAt  TokenHash  IsRevoked
DeviceConfigs                            # 1:1 with Devices
  DeviceId  SystemPrompt  Voice  Model  IdleTimeoutSeconds  EnabledToolsJson
PairingCodes
  Code (8 chars)  OwnerUserId  ExpiresAt  ConsumedAt
Conversations
  Id  DeviceId  StartedAt  EndedAt  EndReason ("tool" | "idle" | "error")
Turns
  Id  ConversationId  Index  Role ("user" | "assistant" | "tool")  Text  ToolName  ToolArgsJson  ToolResultJson  CreatedAt
SystemSettings                           # singleton row
  DefaultVoice  DefaultModel  DefaultSystemPrompt
  • TokenHash = SHA-256 of the device token; plaintext is shown once during pairing and never persisted.
  • EnabledToolsJson is a small JSON array of tool names; no relational table needed.
  • Turns.Index is monotonic per-conversation for ordering.
  • Database.Migrate() runs on startup; SQLite file at /data/assistant.db on a Coolify named volume.

Pairing flow

UI side:

  1. User signs in, clicks Add device, enters a friendly name.
  2. Backend generates an 8-char alphanumeric code from a 32-char alphabet (no 0/O/1/l/I); stores PairingCodes(Code, OwnerUserId, ExpiresAt = NOW + 10 min).
  3. UI shows the code and the one-line installer: 
    curl -fsSL https://<backend>/install.sh | bash
  4. Code stays visible on the device page until consumed or expired; regenerate at any time.

Pi side, first run:

  1. Installer runs.
  2. If ~/assistant/state/config.json exists with a valid device_token, skip pairing and proceed to update path.
  3. Otherwise prompt Pairing code: .
  4. POST https://<backend>/api/pair with { "code", "hostname", "client_version" }.
  5. Backend validates code (exists, unexpired, unconsumed), creates Devices(OwnerUserId = code.OwnerUserId, Name = <user-set-name>, TokenHash = sha256(newToken)), marks code ConsumedAt = NOW, creates default DeviceConfigs row from SystemSettings, returns { "device_id", "device_token", "backend_ws" }.
  6. Installer writes ~/assistant/state/config.json (mode 0600).

Security:

  • Pairing codes are one-shot, 10-min TTL, 8 chars from a 32-char alphabet (~40 bits — fine for the TTL window).
  • Device tokens are 32 bytes random, base64url-encoded, stored only as SHA-256 hash server-side.

Install script

install.sh is served by the backend at GET /install.sh and is fully idempotent.

1.  Resolve BACKEND_URL from the script's own served-from origin (no hardcoding).
2.  apt-get install -y python3-venv libportaudio2 alsa-utils
       (sudo prompt OK; passwordless-sudo per findings.md §E is the recommended setup).
3.  mkdir -p ~/assistant/{code,state}
4.  python3 -m venv ~/assistant/.venv  (if missing)
5.  curl -fsSL $BACKEND_URL/client.tar.gz | tar -xz -C ~/assistant/code
6.  ~/assistant/.venv/bin/pip install -r ~/assistant/code/requirements.txt
7.  ~/assistant/.venv/bin/pip install --no-deps -r ~/assistant/code/requirements-nodeps.txt
       (openwakeword install split per findings.md §3).
8.  If config.json missing → run `~/assistant/.venv/bin/python -m client.pair`,
       which prompts for the code and writes config.json.
9.  Install systemd user unit at ~/.config/systemd/user/assistant.service.
10. systemctl --user daemon-reload
11. systemctl --user enable --now assistant
12. loginctl enable-linger pi   (so the unit runs without an active login).
13. Print status; if pairing happened, print "Paired to <backend> as device <name>".

The systemd user unit runs ~/assistant/.venv/bin/python -m client.main with:

  • Restart=on-failure
  • RestartSec=5
  • Environment=PA_ALSA_PLUGHW=1

Updates

Re-running the same one-liner upgrades the client. Steps 5–7 fetch and reinstall the latest bundle; step 11 restarts the unit. Pairing is skipped because config.json already exists.

curl ... | bash -s -- --reset deletes config.json and forces a fresh pairing.

Backend install endpoints

  • GET /install.sh — returns the shell script with BACKEND_URL baked from the request's X-Forwarded-Host / Host. Public, no auth.
  • GET /client.tar.gz — returns the current client bundle (built at deploy time into wwwroot/). Public, no auth — the code is not a secret; the device token gates the device WS.
  • POST /api/pair — accepts pairing code, returns device token. No prior auth (code is the credential).

The bundle is built during the Docker image build: a stage runs tar czf client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt and copies the result into wwwroot/.

Error handling

Scenario Detection Backend action Client action
Pi loses network WS close / heartbeat miss Close upstream Realtime WS if active, log conversation EndReason="error", mark device offline Exponential backoff reconnect 1→30 s, stay IDLE, no wakeword
Backend restarts Pi WS closes from server side (n/a — process gone) Same backoff reconnect; on reconnect send fresh hello
OpenAI WS drops mid-conversation upstream on_close Send error(code=upstream) + session_ended(reason=error), persist partial transcript Play sleep.wav, return to IDLE
OpenAI rate limit / 429 on connect upstream open fails Don't open session; send error(code=upstream, message=...) instead of session_started Play sleep.wav, IDLE
Tool execution exception (server-side) C# try/catch around ITool.ExecuteAsync Send tool result {"error": "..."} back to OpenAI so the model can recover; log (no client involvement)
Pi-side tool fails (e.g. set_volume amixer error) tool_result.ok=false Forward {"error": "..."} to OpenAI Log and continue
Device token revoked while connected Admin clicks revoke Set IsRevoked=true, close all WS for that device with error(code=auth_revoked, fatal=true) Stop the process (systemd will restart, fail to auth, sleep 60 s, retry — visible in logs)
SQLite write failure EF exception Bubble up to REST as 500; on the hub side, drop the offending log event, keep session running (n/a)
Two simultaneous wakeword fires wake while not IDLE Backend ignores second wake (idempotent within active session) Client also gates locally
Audio buffer overflow on Pi PortAudio callback queue full (n/a) Drop oldest frame, log a warning

Management UI (v1)

React SPA, served from the backend's wwwroot/. Pages:

  • Sign in / sign up (ASP.NET Identity).
  • Devices — list of the signed-in user's devices: name, online indicator, last-seen.
  • Device detail — rename; generate pairing code (with countdown); revoke; per-device config form: system prompt override, voice, model, idle timeout slider, enabled-tools checkboxes; conversation history viewer (paginated, text only).
  • Admin (admin role only) — user list, invite/disable users, global model/voice/prompt defaults.

Out of v1: live conversation tail, audio meters, audio recordings, memory UI, per-device wake.wav/sleep.wav upload (filesystem only for v1; v1.1 adds upload).

Testing strategy

  • Backend unit tests (xUnit): ToolRegistry filtering by enabled set; PairingService (code generation, expiry, single-use); ConversationLog (turn ordering, persistence); auth/role enforcement on the admin endpoints.
  • Backend integration tests (WebApplicationFactory): full pair flow end-to-end against in-memory SQLite; device-hub WS with a fake OpenAI upstream (test double that emits scripted Realtime events).
  • Client unit tests (pytest): state machine transitions (pure-Python, no audio); resample correctness against golden samples; config loading. Audio I/O and wakeword are not unit-tested — already covered by the prototypes and by manual end-to-end.
  • End-to-end manual smoke test (in repo README.md): pair a fresh Pi, say wakeword, ask "what time is it", say "bye", verify conversation row + sleep.wav heard. Run after any change that touches the conversation flow.
  • No load / perf testing.

Deployment (Coolify)

Single Docker image, multi-stage Dockerfile (preparing-dotnet-react-app-for-coolify skill is the reference for the layout):

  1. Frontend build stage — Node, npm ci && npm run build against frontend/.
  2. Backend build stage — .NET SDK, dotnet publish backend/ -c Release -o /publish.
  3. Client bundle stage — tar czf /publish/wwwroot/client.tar.gz -C . client/ requirements.txt requirements-nodeps.txt.
  4. Final stage — mcr.microsoft.com/dotnet/aspnet:9.0, copies /publish, copies SPA dist/ into wwwroot/, sets ASPNETCORE_URLS=http://+:8080, exposes 8080.

Coolify app (provisioned via deploying-to-coolify-via-api):

  • Source: this repo on Gitea.
  • Expose 8080 (Traefik fronts it).
  • SERVICE_FQDN_BACKEND → backend domain.
  • Env vars:
    • OPENAI_API_KEY — secret.
    • ConnectionStrings__Default=Data Source=/data/assistant.db
    • ASPNETCORE_URLS=http://+:8080
  • Named volumes:
    • /data — SQLite file.
    • /keys — ASP.NET data-protection keys (cookies survive container restarts).
  • Deploys triggered fire-and-forget via the Coolify v4 API (per global CLAUDE.md rule).

Repo layout (final)

Assistant/
├── CLAUDE.md
├── README.md                          # smoke-test instructions
├── findings.md                        # prototype-phase findings (kept)
├── tests/                             # prototype throwaways (kept for reference)
│   ├── 01-record-play/
│   └── 02-wakeword/
├── docs/superpowers/
│   ├── specs/
│   │   ├── 2026-06-11-voice-assistant-prototypes-design.md
│   │   └── 2026-06-11-smart-assistant-design.md   # this design
│   └── plans/
│       └── (one or more implementation plans, written next)
├── backend/                           # ASP.NET solution
│   ├── backend.csproj
│   ├── Program.cs
│   ├── Auth/  Devices/  DeviceHub/  Realtime/  Tools/  Conversations/  Install/  Data/
│   └── wwwroot/                       # filled at build: SPA dist + client.tar.gz
├── frontend/                          # React + Vite source
│   ├── package.json
│   └── src/
├── client/                            # Python package
│   ├── requirements.txt               # sounddevice, numpy, scipy, onnxruntime, websockets, ...
│   ├── requirements-nodeps.txt        # openwakeword
│   ├── main.py  audio.py  wakeword.py  session.py  playback.py  state.py  config.py  pair.py  log.py
│   └── systemd/assistant.service
├── Dockerfile
└── deploy.json                        # Coolify config (project/server uuids, domain)

Open items deferred to implementation plans

  • Exact OpenAI Realtime model + voice default (SystemSettings.DefaultModel/DefaultVoice) — pick at implementation time from currently-available choices.
  • Whether wake.wav / sleep.wav playback uses the persistent OutputStream (preferred — same path as session audio) or a separate one-shot — settled in the client plan.
  • ASP.NET data-protection key seeding (initial bootstrap on first run).
  • React UI component library — Tailwind only, or Tailwind + a small headless component lib (Radix / shadcn).
  • Exact mixer control name for set_volume on the LISTENAI device — confirm with amixer scontrols on the Pi during implementation.

Reference paths

  • Prototype spec: docs/superpowers/specs/2026-06-11-voice-assistant-prototypes-design.md
  • Prototype findings: findings.md
  • Pi access: pi@192.168.50.115, password assistant (see CLAUDE.md)
  • Coolify skills: deploying-to-coolify-via-api, preparing-dotnet-react-app-for-coolify
Reference in New Issue View Git Blame Copy Permalink