0. Verify
Done- Payload keys already confirmed in both binaries (strings scan, 2026-07-05)
- Still dump one real Stop payload per CLI: keys existing ≠ keys populated
- Confirm summarizer token readable at runtime
- Remove temp hooks, commit nothing
Fully shipped. Phases 0-6 plus the night shift and a controller v2 live since 2026-07-06.
last_assistant_message as JSON on stdin. No PTY, no ANSI parsing, ever..venv/bin/python by absolute path.What actually runs, and what stays local.
| Layer | What | Notes |
|---|---|---|
| TTS engine | Pocket TTS default, Kokoro-82M optional | Engines sit behind a lazy load / voices / synth registry. Pocket remains the live default. Kokoro exposes 54 voices and joins the controller shootout. Warm bench: Pocket RTF 0.160, Kokoro RTF 0.096. |
| Length limit | None from the engine | Generation is chunk-streamed, so time scales linearly with text and memory stays flat. Arbitrarily long input works; the 400-char condense cap is our taste policy, not a Pocket limit. |
| Condense | ollama / none, switchable live | Ollama runs qwen3.5:4b locally at 11434, streamed with thinking disabled. Table replies send the full redacted response locally so the model can summarize the table instead of losing it. condense_prompt is the only tone knob. |
| Playback | ffplay (atempo) or afplay | ffplay preferred for pitch-preserving rate; afplay fallback. Procedural repo, intent, and build cues are embedded into the same WAV. Optional Spotify and Browser YouTube ducking can fade media down during actual TTS playback, then restore the saved level when playback finishes or is stopped. |
| Runtime | Python 3.11 stdlib + PyTorch | Daemon is stdlib-only (HTTP, threads, queue, SSE). PyTorch rides in under Pocket. No web framework. |
| Runs local | Everything | Synthesis, playback, cloning, config, controller, intent capture, Git evidence, transcript evidence, and condensing stay on-device. |
Scope narrow enough to finish.
| Item | Call | Status |
|---|---|---|
| Product | Speak the final response of every Codex and Claude Code turn, automatically, via hooks. No wrapper command, agents run as normal. | Shipped |
| User flow | Zero. Hooks fire on their own. New input kills speech instantly, scoped to the project you typed in. | Shipped |
| Privacy | Everything stays local. User prompts are redacted and ephemeral; only compact intent metadata persists for restart recovery. | Local |
| Latency | Warm first-audio under 1s. Hook exits within ~100ms. Measured: 0.05s warm first-audio on Pocket. | Passed |
| Voice in | Not built. Hex (com.kitlangton.Hex, Parakeet STT) already covers it. This tool is one-way: agent to speaker. | Non-goal |
| Non-goals | No PTY wrapper, no UI beyond the controller, no voice cloning by default, no native app packaging, no framework, no support for agents I don't use. | Avoid |
The whole thing is a hundred-ish lines.
last_assistant_message.{event, text, ts} to daemon, auto-start it if down, exit. Nothing else./speak, /stop, /health. Table-aware condense + cleanup + synthesis, all async. Project prefixes, .voice.log, lazy load, 10 min idle exit.UserPromptSubmit hits /stop, scoped to the typing project's cwd.Where all the taste lives.
none. Tone comes entirely from condense_prompt.AIUserPromptSubmit in either CLI posts /turn/start with session and turn identity. The daemon acknowledges immediately, snapshots Git asynchronously, and only silences the typing project.RequiredNever break these. Everything else is the model's job.
| Never block the agent | Hook exits within ~100ms: it only POSTs and dies. All cleanup, condensing, synthesis, and playback happen async in the daemon. Hook is invoked as .venv/bin/python by absolute path (warm uv run measured 40-90ms, too knife-edge). |
| Race-safe | Daemon start is bind-winner: both CLIs may fire at once, first to bind :7333 wins, loser just POSTs. Generation counter on utterances: a stale /speak can never resurrect speech after a newer /stop. /stop kills the whole afplay process group. |
| Silence over noise | Any failure (daemon down, API error, malformed payload) means no sound and exit 0. Errors never leak into the agent session. |
| Never speak code | No code, diffs, stack traces, or raw paths, under any condition. |
| Instant interrupt | New user input kills current speech immediately, scoped to the project it came from. |
| SHUT UP | .voice-disabled in the repo root blocks wake-up and speech before anything else runs. Delete it to re-enable, no restart. |
| Local only | Everything runs on-device. Condensing supports Ollama or deterministic none. |
| Stay small | No framework or native app packaging, no tests beyond the bench script. A handful of files. Personal tool, not a product. Amended 2026-07-06: "no config UI" fell deliberately; controller.html is the sanctioned tuning surface, served by the daemon. It is a remote, not a brain: all behavior stays in daemon.py. |
| Secrets | Known secret shapes are redacted before speech processing. No cloud token is required. |
One commit per phase, plain messages.
bench/ for after-the-fact ear test; engine is one swappable constant in daemon.pydaemon.py, localhost :7333, bind-winner start/speak /stop /health, generation IDs/stop kills afplay process grouphook.py, shared by both agents{event, text, ts} only, no logic, no network beyond localhostUserPromptSubmit posts /stopsettings.json, verify real turnconfig.toml, verify real turn/healthcontroller.html served at GET /config.json, hot-appliedrumps status-bar companionHook registration, both CLIs, same script.
{
"hooks": {
"Stop": [{ "hooks": [{
"type": "command",
"command": "$REPO/.venv/bin/python $REPO/hook.py"
}]}],
"UserPromptSubmit": [{ "hooks": [{
"type": "command",
"command": "$REPO/.venv/bin/python $REPO/hook.py"
}]}]
}
}
[[hooks.Stop]] [[hooks.Stop.hooks]] type = "command" command = "$REPO/.venv/bin/python $REPO/hook.py" [[hooks.UserPromptSubmit]] [[hooks.UserPromptSubmit.hooks]] type = "command" command = "$REPO/.venv/bin/python $REPO/hook.py"
Build calls 2026-07-05, post-ship calls 2026-07-06. Not open for relitigation.
| Question | Decision | Why |
|---|---|---|
| Capture method | Hooks only, no PTY fallback | Both CLIs ship last_assistant_message in the Stop payload. PTY-era rationale is dead. |
| Agent order | Claude Code first, Codex second | Older hook system, known-working examples. Codex is a config paste after. |
| Long responses | Local Ollama condense in the daemon | Network work never enters the hook. Setting the provider to none uses the deterministic fallback. |
| Bench order | Pocket, Kokoro, Supertonic | Revised 2026-07-06: Kokoro first after the abstraction, then Supertonic only after the Kokoro shootout works end to end. |
| Voice | Stock preset, calm, slightly fast | Simplest first. Cloning and variants later, maybe never. |
| Daemon lifecycle | Spawn-on-first-hook, 10 min idle exit | A local developer machine may already be running agents and a browser. No forever-resident model. |
| Voice input | Out of scope | Hex already does local Parakeet STT. |
| TTS engine | Pocket default, Kokoro optional | Kokoro shootout shipped. Warm result on the same sentence: Pocket RTF 0.160, Kokoro RTF 0.096. Pocket remains default until a winner is picked. |
| SHUT UP | .voice-disabled file in repo root | Zero config, greppable, no daemon restart. Hook checks it before anything else. |
| Stop scope | cwd-scoped /stop (2026-07-06) | Two projects run concurrently in practice; typing in one must not silence the other. working_cwd covers the condense/synthesis window. |
| Table replies | Raw reply to summarizer, speak the summary | Reading rows aloud is noise, but stripping tables before the model loses context. If the summarizer is unavailable, fall back to the spoken invite. |
| Condense failure | Audible failure line, superseded: first sentence + note (Phase 5) | Audible beat silent, but losing the content entirely was worse than a degraded summary. |
| Project prefix | Rotating In X / From X / Update from X when 2+ projects inside 10 min | The ear needs a namespace when two agents talk; rotation keeps it from sounding robotic. |
| Config UI | controller.html, Phase 6 (2026-07-06) | The taste knobs turned out to be the product. Amends the 2026-07-05 "no config UI" invariant deliberately: the controller tunes, the daemon decides. |
| Voice | Catalog voices + per-agent assignment; cloning plumbing built, gated | The 26-voice catalog ships now. Cloning (drop-a-wav / record-in-browser) works end to end but needs the gated HF weights, so it degrades to an actionable setup message until the one-time auth step is done. |
| Playback engine | ffplay atempo, play on synthesis-complete (2026-07-06) | ffplay atempo changes speed without pitch-shifting (afplay -r shifts). True mid-synthesis PCM streaming was cut: ffplay's pipe exits early once buffered and truncates audio, and the latency win was marginal because condensing keeps replies short. |
| Live UI | SSE stream, not polling (controller v2) | One stdlib text/event-stream faucet pushes speak / now / wave events; the browser's EventSource auto-reconnects. Replaced the 5s poll so blips and the waveform land instantly. |
| Local condense | Ollama with qwen3.5:4b, or none | Ollama runs locally at 11434, streamed with thinking disabled. The default model fits alongside Pocket TTS on the target Mac. |
| Personality prompt | condense_prompt is the sole tone knob (2026-07-10) | The clean/natural/radio speech_style layer was removed: it was a second, redundant tone control sitting on top of the prompt and diluting it. Whatever the user writes into condense_prompt now reaches the model unmodified. |
| Daemon restart | POST /restart self-respawn (2026-07-10) | Editing daemon.py mid-session required a manual pkill + relaunch to pick up code changes; SHUT UP does not restart the process, it only blocks speech. The endpoint spawns a detached delayed relauncher, then hard-exits, so the port is free before the new process binds it. |
Pivot, don't polish.
| Risk | Mitigation |
|---|---|
last_assistant_message present but unpopulated at runtime | Resolved 2026-07-05: real Stop payloads dumped from both CLIs, populated. |
| Pocket TTS fails the ear test | Resolved: passed the gate and the ear, live since 2026-07-05. |
| Concurrent replies drop: single pending slot, two projects finish close together | Log-proven usage pattern: multiple repos can finish agent turns close together. Phase 5: per-project pending map. |
| Un-condensed walls of speech: 10 of 29 speaks were 400-913 chars raw | Phase 5 drops the threshold to 400 so the summarizer fires far more often. |
| First utterance after idle exit | Model reload means the first response after 10 min quiet will miss the 1s target. Accepted tradeoff, not a bug. |
| Summarizer down or slow | Speak the first sentence instead. Never wait, never error. |
| RAM pressure in practice | Drop to Supertonic-3 int8, smallest footprint of the three. |
| Kill Criteria | Action |
|---|---|
| No candidate passes <1s warm | macOS say as stopgap, revisit engines later. |
| Speech feels distracting | Speak only condensed summaries and blocked states. |
| Hook ever slows a turn | Fix immediately or disable. The agent is sacred. |
| Setup gets heavy | Cut back to daemon + hook, two files, nothing else. |
First real week, from .voice.log, 2026-07-05/06.
Aligned 2026-07-06 after the deep review. One-liners, ordered by payoff.
| # | Action | One-liner | Weight |
|---|---|---|---|
| 1 | Condense threshold | Drop MAX_DIRECT_CHARS 1000 to 400 so long replies get summarized instead of narrated for a minute. | high |
| 2 | Token-level stripping | Strip path/URL/extension tokens from sentences instead of deleting the whole line in strip_markdown. | high |
| 3 | Scoped stop, fully | On a matched stop, only clear pending if its cwd matches; keep the other project's queued reply alive. | high |
| 4 | Per-project pending | Replace the single slot with a tiny per-cwd map so concurrent project replies both get spoken, newest-per-project wins. | high |
| 5 | Condense fallback | On summarizer failure, speak first_sentence(text) plus a short failure note instead of losing the content. | med |
| 6 | Cold-boot retry | In hook.py, retry /speak once after ~300ms so the first reply after a full shutdown isn't lost. | med |
| 7 | Notification hook | Add Notification event: speak "needs your input" so the blocked-and-waiting moment is audible. | med |
| 8 | Wav leak fix | Unlink the temp wav in play() when the generation check bails early. | low |
| 9 | Dead code sweep | Remove kill_player, prepare_text, unused urllib.error imports. Keep first_sentence, item 5 revives it. | low |
| 10 | Log cap | Truncate .voice.log when it crosses ~500 lines. | low |
| 11 | Richer /health | Report model-loaded, uptime, and last-spoken timestamp for one-glance debugging. | low |
Aligned 2026-07-06. Three panels, one file, the daemon serves it.
| Panel | What it does | Why it earns its place |
|---|---|---|
| Mixing Desk | The condense threshold as a draggable line over a live histogram of spoken_chars from .voice.log, with a "would have condensed: N%" readout. Below it: idle exit, prefix window, summary timeout, playback rate and volume, and media ducking controls. | Turns the highest-impact knobs from magic numbers into something aimed by eye, against real data. |
| Strip Inspector | Last reply raw beside what survived cleanup, deletions highlighted. Fed by the daemon keeping the last raw + cleaned pair in memory. | The cleaner silently eats sentences today. This is the permanent tuning surface for strip_markdown. |
| Voice Lab | Dropdown of the 26 Pocket catalog voices, an editable audition text box (paste anything, hear any voice read it), per-agent voice assignment (claude / codex / notification), drop-a-wav / record-in-browser clone zone (gated on the HF weights). | The moment plumbing becomes character: the ear knows who is talking without the prefix, and TTS is testable by pasting text. |
GET / on 7333 returns controller.html. Same origin, no CORS, still one self-contained file.CoreGET/POST /config backed by config.json in the repo root, hot-applied, no restart. Daemon constants become defaults.CoreGET /voices lists the catalog; POST /audition speaks one fixed sentence in a chosen voice. Voice states cached so switching stays cheap.Coreosascript plus Browser YouTube page-volume ducking through the companion Chromium extension. Each target saves its current level, fades to duck_target_volume before playback, and restores in _finish(). No global volume, no tab mute, no pause/resume.CorePOST /inbox/return drains the queue into one concise return briefing. Spotify ducking writes a restoration lease before fading and repairs interrupted state at startup or through the watchdog.ShippedGET /log/tail returns recent log entries for the histogram plus the last raw/cleaned pair for the inspector.CorePOST /restart (2026-07-10): daemon acks, logs restart_requested, spawns a detached shell that sleeps 300ms then execs a fresh daemon.py, and the handler calls os._exit(0). Picks up any code or config change without a manual kill. Controller button polls /health and reloads once it answers.CorePhases 5 + 6 workhorse prompt. Paste this, it restates sections 11 and 12 as one instruction block.
Phase 5 refinement + Phase 6 controller for let-there-be-voice. The system is
LIVE: daemon.py
(Pocket TTS, localhost:7333), hook.py wired into Claude Code + Codex via Stop +
UserPromptSubmit hooks. Do not rebuild anything, do not touch the architecture,
no PTY wrapper, no logic back into hook.py beyond what a task below says.
Narrow, surgical changes only.
FIRST: there is good uncommitted work already in the tree (cwd-scoped /stop,
working_cwd tracking, speak_stale guard). Keep it and commit it as its own
commit before starting the tasks.
Field data behind these changes (.voice.log, first real week): 29 speaks,
median 207 chars, 10 speaks over 400 chars spoken raw (max 913, roughly a
minute of narration), condense fired only 3 times with zero failures.
TASKS, in order:
1. Condense threshold: MAX_DIRECT_CHARS 1000 -> 400 in daemon.py.
2. Token-level stripping: in strip_markdown, stop deleting whole lines that
contain file paths, URLs, or filename.ext tokens. Remove the offending
token from the sentence, keep the rest of the sentence. Keep dropping
diff/traceback/code-fence lines whole as today.
3. Scoped stop, fully: in stop_speech, when a stop matches, only clear
pending work whose cwd matches the stopping cwd. A stop from project A
must never wipe project B's queued reply. cwd=None still means stop all.
4. Per-project pending: replace the single pending slot with a small dict
keyed by cwd (None key allowed). Newest wins PER PROJECT; the worker
drains all pending entries. Generation logic must stay airtight: a stale
speak never plays after a newer stop for that cwd. Keep it simple, this
is two concurrent projects in practice, not a job queue.
5. Condense fallback: when the summarizer fails, speak first_sentence(text)
prefixed with a short note ("Summary failed, first line:") instead of
the old failure constant which loses the content. first_sentence already
exists in daemon.py, currently unused.
6. Cold-boot retry: in hook.py, if the /speak POST fails, retry once after
~300ms so the first reply after a full shutdown is not lost. Total worst
case stays well under half a second; the Stop hook can afford it. Do NOT
add retries to /stop.
7. Notification hook: handle hook_event_name == "Notification" in hook.py:
POST /speak with a short fixed line like "<project> needs your input"
(project = basename of cwd). Register the Notification hook for Claude
Code in ~/.claude/settings.json alongside Stop, same command. Skip Codex
if it has no equivalent event.
8. Wav leak: in play(), unlink the temp wav before returning when the
generation check bails early.
9. Dead code: remove kill_player and prepare_text from daemon.py, and the
unused urllib.error imports in both files. KEEP first_sentence (task 5
uses it).
10. Log cap: when .voice.log exceeds ~500 lines, truncate it to the most
recent 250 before appending.
11. /health: include model_loaded (bool), uptime_s, last_spoken_ts in the
JSON response.
PHASE 6 - CONTROLLER (only after all Phase 5 tasks are committed + verified):
controller.html, one self-contained file in the repo, served by the daemon at
GET / on 7333 (same origin, no CORS). House skin: dark grey, border-radius 0,
Inter for text, Share Tech Mono for numbers, compact and tight. No framework,
no build step, no external assets. Plain fetch, no SSE, no websockets.
Daemon side, stdlib only:
- GET / returns controller.html from disk.
- GET /config and POST /config: MAX_DIRECT_CHARS, IDLE_EXIT_S,
PROJECT_WINDOW_S, CONDENSE_TIMEOUT_S, afplay rate + volume, and per-agent
voice assignments become defaults overridden by config.json in the repo
root, hot-applied on POST, no restart. afplay rate/volume apply via
afplay -r / -v at play time.
- GET /voices: the predefined Pocket catalog names
(pocket_tts _ORIGINS_OF_PREDEFINED_VOICES keys).
- POST /audition {voice}: speak one fixed sentence in that voice. Cache
voice states per name so switching stays cheap.
- GET /log/tail: recent .voice.log entries for the histogram, plus the last
raw reply and its cleaned version (keep both in memory after each speak).
Three panels, nothing more:
1. Mixing Desk: the condense threshold as a draggable line over a histogram
of spoken_chars from the log, live "would have condensed: N%" readout,
plus sliders for idle exit, prefix window, summary timeout, playback
rate, volume.
2. Strip Inspector: last reply raw beside what survived cleanup, deletions
highlighted. The tuning surface for strip_markdown.
3. Voice Lab: dropdown of catalog voices with an audition button, per-agent
voice assignment (claude / codex / notification), and a drop-a-wav clone
zone rendered but DORMANT (disabled, with a one-line note) until the
gated HF weights are set up manually. Do NOT attempt hf auth yourself.
The controller is a remote, not a brain: all behavior stays in daemon.py.
Verify Phase 6: open the controller, drag the threshold, confirm config.json
updates and a following long reply respects it without a daemon restart;
audition two voices; confirm the inspector shows the raw/cleaned pair from
the last real turn.
NIGHT SHIFT (only after Phase 6 verifies; this is an unattended overnight
run, so self-check relentlessly and never leave the tree broken):
12. smoke.sh: one script in the repo running the whole verify ritual:
daemon health, hook smoke test from CLAUDE.md, cwd-scoped /stop, two
overlapping project turns, kill switch on then off. Run it after every
commit from here on; if it fails, fix before moving to the next task.
13. Error journal: wrap the daemon worker and request handlers so any
exception lands in .voice.log as {"event":"error","trace":...}. Never
crash the daemon for one bad speak; log and continue.
14. Personality editor: move the summarizer condense system prompt into
config.json (default = the current prompt text), expose it via /config,
add a textarea panel in the controller with a "re-condense last long
reply" button so a prompt change is instantly hearable.
15. Quiet hours: config.json gains quiet_hours {"start":"HH:MM",
"end":"HH:MM"} (absent = disabled). Inside the window /speak is
accepted, logged as {"event":"quiet_skip"}, and stays silent. The kill
switch and /stop are unaffected. Expose the window in the Mixing Desk.
16. Ledger panel: a fourth read-only controller panel from .voice.log:
talk-time today (estimate from spoken_chars at ~15 chars/sec), condense
rate, table skips, per-project utterance lanes over time. No new
endpoint beyond /log/tail growing a limit param.
17. Notification parity, both CLIs: first verify the Claude Code
Notification hook from task 7 actually fires on a real permission
prompt and speaks. Then investigate whether Codex CLI has an equivalent
event (notify config, hooks); wire it identically if yes. Record what
you found either way in the implementation notes.
18. Notification priority: a "needs your input" line queue-jumps. It never
waits behind a long reply mid-playback in another project: it speaks
next, ahead of any pending replies. Do not cut off an utterance that is
already playing for the SAME project the notification belongs to;
interrupt other-project playback only if implementation stays simple,
otherwise just jump the queue.
19. Config armor: a malformed or hand-mangled config.json must never
silence the system. Validate on load and on POST /config; on any bad
field fall back to that field's default, log
{"event":"config_error","field":...}, keep speaking. The failure mode
of a bad write is defaults, never mystery silence.
20. Transcript fallback: on Stop, if last_assistant_message is empty but
transcript_path is present, parse the transcript for the last
assistant entry and speak that. This was the original risk-table
fallback, never built. Log {"event":"transcript_fallback"} when used.
21. voicectl: a tiny ./voice shell script in the repo: status (pretty
/health), chill (stop current speech), shutup (create .voice-disabled), unshutup (remove it),
say "text" (POST /speak). Stdlib/curl-level simplicity, no deps.
22. Temperature knob: expose Pocket's TTS temperature (default 0.7) in
config.json and as a Mixing Desk slider. It is passed at generation
time; confirm pocket_tts generate_audio_stream accepts it and wire it
through, else note the limitation in the implementation notes and skip.
23. Docs true-up, last task of the night: update CLAUDE.md to post-run
reality (endpoints, config.json, controller, smoke.sh, voicectl, quiet
hours), and stamp the running git rev into /health so "which daemon
build is running" is never a guess.
End the pass by writing short implementation notes: what landed, what failed,
what you skipped and why, plus anything surprising in the logs. Plain prose,
short.
INVARIANTS, never break these ("no config UI" was amended 2026-07-06 to
permit exactly this controller, nothing more):
- Hook exits fast and never blocks the agent turn; silence over noise: any
failure means no sound and exit 0, never leak errors into the session.
- Never speak code, diffs, stack traces, or raw file paths.
- New input in a project kills that project's speech instantly; stale speech
never resurrects after an interrupt.
- SHUT UP: .voice-disabled in the repo root blocks everything.
- Everything local except the optional summarizer condense call; only the daemon
touches the network or the token.
- Keep it small: no framework, no native app packaging, no tests beyond the bench
script. The controller is the only UI, and it stays one file. Personal
tool, not a product.
VERIFY when done: run the hook smoke test from CLAUDE.md, then one real
Claude Code turn and one real Codex turn, then two overlapping turns from two
different project directories to prove per-project pending and scoped stop
actually behave.
Commits: the pre-existing scoped-stop work first, then one commit per task or
sensible small groups, plain messages. Phase 6 lands as its own commit(s)
only after Phase 5 is verified end to end, and the Night Shift only after
Phase 6 does. Run smoke.sh before every commit once it exists.
Partly shipped 2026-07-06. Pocket default, Kokoro wired and proven, Supertonic still queued.
| # | Feature | What it does |
|---|---|---|
| 0 | Engine abstraction | Shipped Uniform lazy engine registry. Pocket behavior preserved; Kokoro slots in as a sibling. |
| 2 | Shootout mode | Shipped Paste one line, every installed engine synthesizes it, play each clip on demand through the replay ring. |
| 3 | Metrics scoreboard | Shipped Daemon returns RTF, TTFA, synth time, duration, RSS, and clip id per engine. |
| 5 | Blind A/B/C | Shipped Clips randomize into unlabeled buttons, then reveal. |
| 6 | Per-engine voices | Shipped Voice Lab follows the active engine catalog: Pocket 26, Kokoro 54. |
| 8 | Winner + cleanup | Shipped "Keep" sets the default and prints cleanup commands, without running removals. |
| 9 | Readiness cards | Shipped Per engine installed / loaded / RAM cards, with install commands when missing. |
GET /engines (installed / loaded / ram), POST /engine (set active), POST /bench {text} (synth on all installed, return clips + metrics). Audition and speak gain an optional engine.CorePaste this to the workhorse. Restates section 14 as one instruction block.
Phase 9 for let-there-be-voice: turn the controller into a TTS shootout bench
so I can compare three engines, learn, pick the best, and remove the losers.
The system is LIVE and must keep working throughout. Read build-packet.html
section 14 for the full spec. Do not break the live voice path: Pocket stays
the default engine until I pick a winner in the UI.
FOUNDATION FIRST (task 0, its own commit):
Refactor synthesis behind an engine abstraction. Today ENGINE = "pocket" and
speak() calls Pocket directly. Introduce a small registry where each engine
exposes the same interface: load() (lazy), voices() -> list of names,
synth(text, voice) -> yields int16 PCM chunks at the engine's sample_rate.
Move today's Pocket code behind it UNCHANGED in behavior; the live path,
smoke.sh, cloning, and the controller must all still pass exactly as now.
config gains "engine" (default "pocket") validated against installed engines.
KOKORO SECOND (prove it before touching Supertonic):
Add Kokoro-82M as an engine (kokoro pip package, Apache-2.0, PyTorch, happy on
MPS). Expose its ~54 voices via voices(). Synth to int16 PCM chunks. If the
package is not installed, the engine reports "not installed" with the pip/uv
command, never crashes. Once Kokoro synthesizes and plays through the existing
pipeline, wire the controller bench (below) and stop. Do NOT add Supertonic in
the same run.
CONTROLLER BENCH (features 2,3,5,6,8,9 from section 14):
- GET /engines: [{name, installed, loaded, rss_mb}]. POST /engine {name}: set
active, hot-applied, saved to config. POST /bench {text, voice?}: synthesize
the text on every installed engine, return per-engine {clip_id, rtf,
ttfa_s, synth_s, duration_s, rss_mb}. Audition/speak take an optional engine.
- Readiness cards (9): per-engine installed / loaded / RAM, install command
when missing.
- Shootout (2): a text box, a "compare" button, plays each engine's clip on
demand, reuse the replay ring for the clips.
- Metrics scoreboard (3): the /bench numbers in a live per-engine table. RTF =
synth_s / duration_s, measured in the daemon (resource.getrusage for RAM,
perf_counter for timing). RTF below 1.0 = faster than realtime.
- Blind A/B/C (5): play the three clips unlabeled in random order, let me
score each, then reveal which engine was which.
- Per-engine voices (6): the Voice Lab dropdowns and catalog reflect the
ACTIVE engine's voices, not a shared list.
- Winner + cleanup (8): a "keep this engine" control that sets the default and
prints the uninstall / file-removal steps for the other engines. Do NOT run
the removal yourself; just show the commands.
SUPERTONIC LAST (only after the Kokoro shootout works end to end):
Add Supertonic-3 via sherpa-onnx as a third engine behind the same interface,
then it joins the shootout automatically.
INVARIANTS: the live voice path never regresses; Pocket default until I pick;
engines lazy-load and idle-exit as today; stdlib daemon plus the engine
packages, no web framework; controller stays one self-contained file and a
remote, not a brain; run smoke.sh before every commit and keep it green.
VERIFY: smoke.sh green, one real Claude turn still speaks via Pocket, then a
bench run comparing Pocket vs Kokoro on the same sentence with real RTF
numbers. One commit per meaningful step, plain messages. When done, update
CLAUDE.md and this packet per the docs-sweep note.