Assistant/README.md

# Smart Assistant

Voice assistant on a Raspberry Pi talking to the OpenAI Realtime API via an
ASP.NET backend deployed on Coolify.

See `docs/superpowers/specs/2026-06-11-smart-assistant-design.md` for the full
design. Implementation plans:

- Plan 1 — `docs/superpowers/plans/2026-06-11-smart-assistant-backend-foundation.md`
  (Identity, devices, pairing, install endpoints).
- Plan 2 — `docs/superpowers/plans/2026-06-11-smart-assistant-device-hub-realtime-tools.md`
  (device WebSocket hub, OpenAI Realtime relay, tools registry).

## Repo layout

- `backend/` — ASP.NET 9 backend (Identity, Devices, Pairing, Install).
- `backend.tests/` — xUnit integration + unit tests via `WebApplicationFactory`.
- `client/` — Python client. **Plan 1 ships placeholders only**; the full
  wakeword + Realtime client lands in Plan 3.
- `Dockerfile` + `docker-compose.yml` — multi-stage build; produces one image
  with the published ASP.NET app + a bundled `client.tar.gz` in
  `wwwroot/client.tar.gz`.
- `deploy.json` — Coolify config (gitignored; contains the OpenAI key).
- `docs/superpowers/` — specs and plans.

## Local development

```sh
dotnet test                              # all backend tests
dotnet run --project backend             # starts on http://localhost:5252
```

Hit `http://localhost:5252/health` to verify it's up.

## Smoke test against a live deploy

The full pairing flow can be exercised with `curl` — no UI needed yet.

```sh
DOMAIN="https://assistant.volcanic.tes.gd"          # or your domain
COOKIE=$(mktemp)

# Register the first user (becomes Admin automatically) and sign in
curl -sf -c "$COOKIE" -X POST "$DOMAIN/api/auth/register" \
  -H 'content-type: application/json' \
  -d '{"email":"you@example.com","password":"Passw0rd!"}'
curl -sf -c "$COOKIE" -b "$COOKIE" -X POST "$DOMAIN/api/auth/login" \
  -H 'content-type: application/json' \
  -d '{"email":"you@example.com","password":"Passw0rd!"}'

# Generate a pairing code (auth required)
CODE=$(curl -sf -c "$COOKIE" -b "$COOKIE" -X POST "$DOMAIN/api/pair-code" \
  -H 'content-type: application/json' \
  -d '{"name":"kitchen"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["code"])')
echo "code=$CODE"

# Pair as a device (public endpoint) and capture the token
curl -sf -X POST "$DOMAIN/api/pair" \
  -H 'content-type: application/json' \
  -d "{\"code\":\"$CODE\",\"hostname\":\"smoke\",\"client_version\":\"0.1\"}" \
  | python3 -m json.tool

# Verify the install script and client bundle are served
curl -sf "$DOMAIN/install.sh" | head -5
curl -sIf "$DOMAIN/client.tar.gz"
```

Expected: `{"ok":true}` on `/health`, JSON with `device_id`/`device_token`/`backend_ws`
from `/api/pair`, the bash shebang from `/install.sh`, and a `200 OK` with
`Content-Type: application/gzip` for the bundle.

> **Plan 1 ships a placeholder Python bundle** — running the bundled
> `client.main` will exit with the placeholder message. Plan 3 replaces it
> with the real wakeword + Realtime client.

## Deploying

Use the Coolify v4 API. The `deploy.json` file (gitignored) carries the
project + server UUIDs and the OpenAI key.

```sh
APP=$(python3 -c "import json; print(json.load(open('deploy.json'))['app']['uuid'])")
curl -sf -X POST "$COOLIFY_URL/api/v1/deploy?uuid=$APP" \
  -H "Authorization: Bearer $COOLIFY_KEY"
```

Fire-and-forget; the API returns a `deployment_uuid` immediately. Don't poll
(per the global Coolify rule in `CLAUDE.md`).

## Plan 2: device hub + Realtime relay

- `wss://<domain>/device` — bearer-token WebSocket. Pair a device first to get a token.
- Handshake: client sends `{type:"hello", device_id, client_version}`, backend
  replies `hello_ack` with per-device config (voice, model, system prompt, idle
  timeout, enabled tools).
- Heartbeat: client sends `{type:"ping"}` every 15 s, backend replies `pong`
  and updates `Devices.LastSeenAt`.
- Wake: `{type:"wake"}` opens a Realtime session against OpenAI; backend emits
  `session_started` (with `conversation_id`), forwards `response.audio.delta`
  as binary frames to the device, persists `response.audio_transcript.done` as
  an assistant turn, persists `conversation.item.input_audio_transcription.completed`
  as a user turn, and emits `assistant_done` after each `response.done`.
- Tools shipped (per-device toggle via `DeviceConfig.EnabledToolsJson`):
  - `get_current_time` (server-side, returns `{"now":"<ISO-UTC>"}`),
  - `end_session` (closes the session after the current `response.done`
    completes; emits `session_ended(reason="tool")`),
  - `set_volume` (round-trips a `tool_call`/`tool_result` envelope to the Pi;
    the Pi side will run `amixer` in Plan 3).
- Idle timeout: server-side. With no `input_audio_buffer.speech_started` upstream
  event for `IdleTimeoutSeconds`, the relay emits `session_ended(reason="idle")`.

The Python Pi client (Plan 3) is the production consumer of this surface. A
quick handshake-only smoke against the live deploy:

```sh
DOMAIN="https://assistant.volcanic.tes.gd"
# Reuse the curl recipe above to log in, generate a pair-code, and pair
# a "smoke" device — capture $TOKEN from the /api/pair response.
python3 - <<PY
import asyncio, json, websockets
URL, TOKEN = "wss://assistant.volcanic.tes.gd/device", "$TOKEN"
async def main():
    async with websockets.connect(
        URL, additional_headers={"Authorization": f"Bearer {TOKEN}"}
    ) as ws:
        await ws.send(json.dumps({
            "type":"hello","device_id":"00000000-0000-0000-0000-000000000000",
            "client_version":"0.1"}))
        print("ack:", json.loads(await ws.recv())["type"])
        await ws.send(json.dumps({"type":"ping"}))
        print("pong:", json.loads(await ws.recv())["type"])
asyncio.run(main())
PY
```

Don't send `wake` from the smoke — it would open a real OpenAI session and
burn budget. Plan 3 exercises that end-to-end.

## Plan 3: Pi client

The Python client lives in `client/`. It is shipped to the Pi inside
`client.tar.gz` (built into the backend Docker image) and installed by
`install.sh`. Modules:

- `main.py` — entry point. Sets `PA_ALSA_PLUGHW=1` before importing
  sounddevice. Wires audio, state, wakeword, session, and playback.
- `state.py` — `IDLE → WAKE_PENDING → LISTENING ↔ ASSISTANT_SPEAKING →
  IDLE_PENDING → IDLE`. `wakeword_enabled` is true only in `IDLE`;
  `uplink_enabled` is true only in `LISTENING`.
- `audio.py` — USB device discovery, persistent 24 kHz/mono/int16 InputStream
  and OutputStream, and the 24 → 16 kHz resample for the wakeword feed.
- `wakeword.py` — openwakeword `alexa` model, threshold + cooldown.
- `playback.py` — single worker thread that owns the OutputStream and
  implements `set_volume` as a software gain (the USB Speaker Phone has no
  hardware playback volume control; only a `PCM Playback Switch`).
- `session.py` — backend WS client, dispatch, tool round-trip,
  exponential-backoff reconnect.
- `config.py` — `~/assistant/state/config.json` (mode 0600).
- `pair.py` — `python -m client.pair --backend <url> [--code <code>]`.

### Updating the Pi

For a fast iteration loop:

```sh
sshpass -p 'assistant' rsync -az --delete --exclude __pycache__ --exclude tests \
  client/ pi@192.168.50.115:assistant/code/client/
sshpass -p 'assistant' ssh pi@192.168.50.115 'systemctl --user restart assistant'
sshpass -p 'assistant' ssh pi@192.168.50.115 \
  'journalctl --user -u assistant -n 50 --no-pager'
```

For a production update (after a backend deploy):

```sh
sshpass -p 'assistant' ssh pi@192.168.50.115 \
  'curl -fsSL https://assistant.volcanic.tes.gd/install.sh | bash'
```

### Local tests

```sh
python3 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt requests websockets numpy scipy
.venv/bin/python -m pytest client/tests
```

Hardware-touching modules (`audio.py` streams, `wakeword.py`, `playback.py`,
`session.run_forever`) are smoke-tested via the live deploy. Deterministic
modules (`state.py`, `config.py`, `pair.py`, the resample helper, the session
dispatcher) are covered by the pytest suite.

### End-to-end smoke

With the unit running on the Pi:

```sh
sshpass -p 'assistant' ssh pi@192.168.50.115 'journalctl --user -u assistant -f'
```

…then say `alexa` near the Speaker Phone. Expected log sequence: `wakeword
fired`, `state IDLE -> WAKE_PENDING`, `session_started conversation_id=…`,
`state WAKE_PENDING -> LISTENING`. Speak a question; the assistant replies
through the speaker (`state LISTENING -> ASSISTANT_SPEAKING`, then
`assistant_done`). Say "bye" → `session_ended reason=tool`, then `state
LISTENING -> IDLE_PENDING -> IDLE`. The conversation row + turns are visible
via the deployed backend's DB (admin UI in Plan 4).

## What's next

- **Plan 4** — React management UI for the admin dashboard, per-device
  config editor, and conversation history viewer.