Smart Assistant

Voice assistant on a Raspberry Pi talking to the OpenAI Realtime API via an ASP.NET backend deployed on Coolify.

See docs/superpowers/specs/2026-06-11-smart-assistant-design.md for the full design. Implementation plans:

• Plan 1 — docs/superpowers/plans/2026-06-11-smart-assistant-backend-foundation.md (Identity, devices, pairing, install endpoints).
• Plan 2 — docs/superpowers/plans/2026-06-11-smart-assistant-device-hub-realtime-tools.md (device WebSocket hub, OpenAI Realtime relay, tools registry).

Repo layout

• backend/ — ASP.NET 9 backend (Identity, Devices, Pairing, Install).
• backend.tests/ — xUnit integration + unit tests via WebApplicationFactory.
• client/ — Python client. Plan 1 ships placeholders only; the full wakeword + Realtime client lands in Plan 3.
• Dockerfile + docker-compose.yml — multi-stage build; produces one image with the published ASP.NET app + a bundled client.tar.gz in wwwroot/client.tar.gz.
• deploy.json — Coolify config (gitignored; contains the OpenAI key).
• docs/superpowers/ — specs and plans.

Local development

dotnet test                              # all backend tests
dotnet run --project backend             # starts on http://localhost:5252

Hit http://localhost:5252/health to verify it's up.

Smoke test against a live deploy

The full pairing flow can be exercised with curl — no UI needed yet.

DOMAIN="https://assistant.volcanic.tes.gd"          # or your domain
COOKIE=$(mktemp)

# Register the first user (becomes Admin automatically) and sign in
curl -sf -c "$COOKIE" -X POST "$DOMAIN/api/auth/register" \
  -H 'content-type: application/json' \
  -d '{"email":"you@example.com","password":"Passw0rd!"}'
curl -sf -c "$COOKIE" -b "$COOKIE" -X POST "$DOMAIN/api/auth/login" \
  -H 'content-type: application/json' \
  -d '{"email":"you@example.com","password":"Passw0rd!"}'

# Generate a pairing code (auth required)
CODE=$(curl -sf -c "$COOKIE" -b "$COOKIE" -X POST "$DOMAIN/api/pair-code" \
  -H 'content-type: application/json' \
  -d '{"name":"kitchen"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["code"])')
echo "code=$CODE"

# Pair as a device (public endpoint) and capture the token
curl -sf -X POST "$DOMAIN/api/pair" \
  -H 'content-type: application/json' \
  -d "{\"code\":\"$CODE\",\"hostname\":\"smoke\",\"client_version\":\"0.1\"}" \
  | python3 -m json.tool

# Verify the install script and client bundle are served
curl -sf "$DOMAIN/install.sh" | head -5
curl -sIf "$DOMAIN/client.tar.gz"

Expected: {"ok":true} on /health, JSON with device_id/device_token/backend_ws from /api/pair, the bash shebang from /install.sh, and a 200 OK with Content-Type: application/gzip for the bundle.

Plan 1 ships a placeholder Python bundle — running the bundled client.main will exit with the placeholder message. Plan 3 replaces it with the real wakeword + Realtime client.

Deploying

Use the Coolify v4 API. The deploy.json file (gitignored) carries the project + server UUIDs and the OpenAI key.

APP=$(python3 -c "import json; print(json.load(open('deploy.json'))['app']['uuid'])")
curl -sf -X POST "$COOLIFY_URL/api/v1/deploy?uuid=$APP" \
  -H "Authorization: Bearer $COOLIFY_KEY"

Fire-and-forget; the API returns a deployment_uuid immediately. Don't poll (per the global Coolify rule in CLAUDE.md).

Plan 2: device hub + Realtime relay

• wss://<domain>/device — bearer-token WebSocket. Pair a device first to get a token.
• Handshake: client sends {type:"hello", device_id, client_version}, backend replies hello_ack with per-device config (voice, model, system prompt, idle timeout, enabled tools).
• Heartbeat: client sends {type:"ping"} every 15 s, backend replies pong and updates Devices.LastSeenAt.
• Wake: {type:"wake"} opens a Realtime session against OpenAI; backend emits session_started (with conversation_id), forwards response.audio.delta as binary frames to the device, persists response.audio_transcript.done as an assistant turn, persists conversation.item.input_audio_transcription.completed as a user turn, and emits assistant_done after each response.done.
• Tools shipped (per-device toggle via DeviceConfig.EnabledToolsJson):
  • get_current_time (server-side, returns {"now":"<ISO-UTC>"}),
  • end_session (closes the session after the current response.done completes; emits session_ended(reason="tool")),
  • set_volume (round-trips a tool_call/tool_result envelope to the Pi; the Pi side will run amixer in Plan 3).
• Idle timeout: server-side. With no input_audio_buffer.speech_started upstream event for IdleTimeoutSeconds, the relay emits session_ended(reason="idle").

The Python Pi client (Plan 3) is the production consumer of this surface. A quick handshake-only smoke against the live deploy:

DOMAIN="https://assistant.volcanic.tes.gd"
# Reuse the curl recipe above to log in, generate a pair-code, and pair
# a "smoke" device — capture $TOKEN from the /api/pair response.
python3 - <<PY
import asyncio, json, websockets
URL, TOKEN = "wss://assistant.volcanic.tes.gd/device", "$TOKEN"
async def main():
    async with websockets.connect(
        URL, additional_headers={"Authorization": f"Bearer {TOKEN}"}
    ) as ws:
        await ws.send(json.dumps({
            "type":"hello","device_id":"00000000-0000-0000-0000-000000000000",
            "client_version":"0.1"}))
        print("ack:", json.loads(await ws.recv())["type"])
        await ws.send(json.dumps({"type":"ping"}))
        print("pong:", json.loads(await ws.recv())["type"])
asyncio.run(main())
PY

Don't send wake from the smoke — it would open a real OpenAI session and burn budget. Plan 3 exercises that end-to-end.

Plan 3: Pi client

The Python client lives in client/. It is shipped to the Pi inside client.tar.gz (built into the backend Docker image) and installed by install.sh. Modules:

• main.py — entry point. Sets PA_ALSA_PLUGHW=1 before importing sounddevice. Wires audio, state, wakeword, session, and playback.
• state.py — IDLE → WAKE_PENDING → LISTENING ↔ ASSISTANT_SPEAKING → IDLE_PENDING → IDLE. wakeword_enabled is true only in IDLE; uplink_enabled is true only in LISTENING.
• audio.py — USB device discovery, persistent 24 kHz/mono/int16 InputStream and OutputStream, and the 24 → 16 kHz resample for the wakeword feed.
• wakeword.py — openwakeword alexa model, threshold + cooldown.
• playback.py — single worker thread that owns the OutputStream and implements set_volume as a software gain (the USB Speaker Phone has no hardware playback volume control; only a PCM Playback Switch).
• session.py — backend WS client, dispatch, tool round-trip, exponential-backoff reconnect.
• config.py — ~/assistant/state/config.json (mode 0600).
• pair.py — python -m client.pair --backend <url> [--code <code>].

Updating the Pi

For a fast iteration loop:

sshpass -p 'assistant' rsync -az --delete --exclude __pycache__ --exclude tests \
  client/ pi@192.168.50.115:assistant/code/client/
sshpass -p 'assistant' ssh pi@192.168.50.115 'systemctl --user restart assistant'
sshpass -p 'assistant' ssh pi@192.168.50.115 \
  'journalctl --user -u assistant -n 50 --no-pager'

For a production update (after a backend deploy):

sshpass -p 'assistant' ssh pi@192.168.50.115 \
  'curl -fsSL https://assistant.volcanic.tes.gd/install.sh | bash'

Local tests

python3 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt requests websockets numpy scipy
.venv/bin/python -m pytest client/tests

Hardware-touching modules (audio.py streams, wakeword.py, playback.py, session.run_forever) are smoke-tested via the live deploy. Deterministic modules (state.py, config.py, pair.py, the resample helper, the session dispatcher) are covered by the pytest suite.

End-to-end smoke

With the unit running on the Pi:

sshpass -p 'assistant' ssh pi@192.168.50.115 'journalctl --user -u assistant -f'

…then say alexa near the Speaker Phone. Expected log sequence: wakeword fired, state IDLE -> WAKE_PENDING, session_started conversation_id=…, state WAKE_PENDING -> LISTENING. Speak a question; the assistant replies through the speaker (state LISTENING -> ASSISTANT_SPEAKING, then assistant_done). Say "bye" → session_ended reason=tool, then state LISTENING -> IDLE_PENDING -> IDLE. The conversation row + turns are visible via the deployed backend's DB (admin UI in Plan 4).

What's next

• Plan 4 — React management UI for the admin dashboard, per-device config editor, and conversation history viewer.