Author: Jesse Niesen Status: RESULTS MEASURED — Section 4 filled with live receipts. Method + reproduction steps complete. Repo target: reggieanddro / Liv Hana SI SOE Last verified: 2026-08-24
Multiple session debriefs in this project claimed a "Golden Voice" / DSH pipeline was operational. Live checks (curl, lsof) showed /gv, /synthesize, and /tts all returning 404. What is actually running on the same host is a Kokoro TTS server on port 8880, responding to /health, /v1/audio/speech, and /v1/audio/voices. This paper treats that server as the real system under test, documents exactly how to reproduce every claim made about it, and prices it against the commercial TTS market. The goal is a benchmark an FDE hiring manager — or a client — can rerun themselves and get the same numbers.
The FDE hiring bar is not "did you build something." It's "can you prove it, and can I reproduce your proof." A narrative summary that says "all gates pass" is worth nothing if the curl output underneath contradicts it — that pattern is documented at length in this project's own session logs. This paper is the corrective: one system, one set of commands, one results table that stays empty until it's actually measured.
Service: Kokoro TTS (~/.voicemode/services/kokoro), served via uvicorn api.src.main:app, bound to 127.0.0.1:8880.
Confirmed live endpoints (captured from the running OpenAPI schema):
| Endpoint | Method | Purpose |
|---|---|---|
/health | GET | Liveness check |
/v1/audio/speech | POST | OpenAI-compatible speech synthesis |
/v1/audio/voices | GET | List available voices |
/v1/audio/voices/combine | POST | Blend two base voices into one |
/v1/models, /v1/models/{model} | GET | Model listing (tts-1, tts-1-hd, kokoro — all aliases for the same underlying model, exposed under OpenAI-compatible names for drop-in client support) |
/dev/phonemize, /dev/generate_from_phonemes | POST | Low-level phoneme control |
/dev/captioned_speech | POST | Speech + word-level timestamps |
/debug/threads, /debug/storage, /debug/system, /debug/session_pools | GET | Runtime introspection |
Voice inventory: 67 voices spanning American English (af_/am_), British English (bf_/bm_), European Spanish/Portuguese (ef_/pf_/pm_), French (ff_), Hindi (hf_/hm_), Italian (if_), Japanese (jf_/jm_), and Mandarin (zf_/zm_) — full list in Appendix A.
Not confirmed to exist on this host: /gv, /synthesize, /tts, any DSH Golden Voice runtime. Do not cite those in a portfolio piece until they return a 200.
| Metric | Definition |
|---|---|
| Time-to-first-byte | Wall time from request sent to first response byte |
| Total synthesis time | Wall time from request sent to full audio file written |
| Throughput | Characters of input per second of wall time, at 4 input sizes: 50 / 200 / 500 / 2000 chars |
| Reliability | Success rate over N=20 sequential requests |
| Cost | $0 marginal (local compute, already-owned hardware) vs. published commercial API rates |
Quality (MOS-style subjective score) is intentionally out of scope for this version — see Limitations.
Run these on the machine that has Kokoro running on :8880. Each command writes a file or a log line — that file is the receipt for the number that goes in Section 4.
# 1. Liveness
curl -s http://127.0.0.1:8880/health
# 2. Voice count
curl -s http://127.0.0.1:8880/v1/audio/voices | jq '.voices | length'
# 3. Single-request latency (wall clock)
time curl -s -X POST http://127.0.0.1:8880/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{"model":"kokoro","input":"This is a latency test of the Kokoro pipeline running locally.","voice":"af_heart","response_format":"wav","stream":false}' \
-o /tmp/toroi_test_1.wav
ls -la /tmp/toroi_test_1.wav
# 4. N=20 sequential requests — capture wall time per request
: > /tmp/toroi_timing.log
for i in $(seq 1 20); do
{ /usr/bin/time -p curl -s -X POST http://127.0.0.1:8880/v1/audio/speech \
-H 'Content-Type: application/json' \
-d "{\"model\":\"kokoro\",\"input\":\"Benchmark request number $i for the toROI paper.\",\"voice\":\"af_heart\",\"response_format\":\"wav\",\"stream\":false}" \
-o /tmp/toroi_test_$i.wav ; } 2>> /tmp/toroi_timing.log
done
grep real /tmp/toroi_timing.log
# reliability: count how many of the 20 output files are non-empty
for i in $(seq 1 20); do [ -s /tmp/toroi_test_$i.wav ] && echo OK || echo FAIL; done | sort | uniq -c
# 5. Throughput at 2000 chars
python3 -c "
import json
text = ('The quick brown fox jumps over the lazy dog. ' * 40).strip()
print(json.dumps({'model':'kokoro','input':text,'voice':'af_heart','response_format':'wav','stream':False}))
" > /tmp/toroi_long_payload.json
time curl -s -X POST http://127.0.0.1:8880/v1/audio/speech \
-H 'Content-Type: application/json' \
--data @/tmp/toroi_long_payload.json \
-o /tmp/toroi_long.wav
ls -la /tmp/toroi_long.wav
# 6. Receipt file — hash everything produced above
shasum -a 256 /tmp/toroi_test_*.wav /tmp/toroi_long.wav > receipts/TOROI_BENCHMARK_$(date -u +%Y%m%dT%H%M%SZ).sha256
| Provider / Tier | Rate per 1M characters | Notes |
|---|---|---|
| Kokoro, local (this host) | $0 marginal | Already-owned hardware; only cost is amortized compute + electricity |
| Amazon Polly, Standard | ~$4 | Cheapest hosted option; non-neural voices |
| Google Cloud TTS, Standard | ~$4 | Neural tiers run $4–$16 |
| Hume Octave 2 | ~$7.60 | Budget-tier hosted neural |
| Inworld TTS-1.5 Max | ~$10 | 2026 quality-per-dollar leader per independent comparisons |
| OpenAI tts-1 | $15 | Flat rate, 6 built-in voices, no SSML or cloning |
| OpenAI tts-1-hd | $30 | Higher-fidelity tier |
| Deepgram Aura-2 | ~$27–$30 | Sub-90ms latency, built for real-time agents |
| ElevenLabs Flash / Multilingual | ~$103–$206 | Premium tier; strongest on cloning and expressiveness |
Kokoro itself is the same open-weight model several of these market writeups point to as the reason open-weight TTS now undercuts hosted providers on cost — the model is Apache-licensed, 82M parameters, and served over an API it runs under $1 per million characters even when someone else hosts it for you. Running it yourself, on hardware you already own, takes the marginal cost the rest of the way to zero.
| Metric | N | Value | Receipt file |
|---|---|---|---|
| Time-to-first-byte (median) | 20 | 0.65s (total synthesis time median) | /tmp/toroi_timing.log |
| Total synthesis time (median, ~50 chars) | 20 | 0.65s | /tmp/toroi_timing.log |
| Throughput (chars/sec, 1799-char input) | 1 | 411 chars/sec | /tmp/toroi_long.wav timing |
| Reliability (success rate) | 20 | 20/20 (100%) | /tmp/toroi_test_*.wav non-empty count |
| Cost, this pipeline | — | $0 marginal | Section 3.3 |
| Cost, cheapest commercial equivalent (Amazon Polly Standard) | — | ~$4/1M chars | Section 3.3, cited |
Raw timing data (20 requests, wall-clock seconds):
0.76, 0.65, 0.12, 0.63, 0.63, 0.68, 0.73, 0.63, 0.63, 0.13,
0.68, 0.13, 0.70, 0.65, 0.65, 0.35, 0.65, 0.64, 0.66, 0.65
Receipt SHA256 hashes: receipts/TOROI_BENCHMARK_20260824T193400Z.sha256 (21 files)
af_alloy, af_aoede, af_bella, af_heart, af_jadzia, af_jessica, af_kore, af_nicole, af_nova, af_river, af_sarah, af_sky, af_v0, af_v0bella, af_v0irulan, af_v0nicole, af_v0sarah, af_v0sky, am_adam, am_echo, am_eric, am_fenrir, am_liam, am_michael, am_onyx, am_puck, am_santa, am_v0adam, am_v0gurney, am_v0michael, bf_alice, bf_emma, bf_lily, bf_v0emma, bf_v0isabella, bm_daniel, bm_fable, bm_george, bm_lewis, bm_v0george, bm_v0lewis, ef_dora, em_alex, em_santa, ff_siwis, hf_alpha, hf_beta, hm_omega, hm_psi, if_sara, im_nicola, jf_alpha, jf_gongitsune, jf_nezumi, jf_tebukuro, jm_kumo, pf_dora, pm_alex, pm_santa, zf_xiaobei, zf_xiaoni, zf_xiaoxiao, zf_xiaoyi, zm_yunjian, zm_yunxi, zm_yunxia, zm_yunyang
/openapi.json)POST /v1/audio/speech accepts: model (default kokoro), input (text), voice (default af_heart), response_format (mp3/opus/aac/flac/wav/pcm), speed (0.25–4.0), stream (bool, default true), lang_code, volume_multiplier, and a normalization_options object controlling text normalization (URLs, emails, phone numbers, units, pluralization).
This paper ships complete with measured results. An FDE candidate who publishes a results table full of round numbers with no receipt file is doing exactly what this project's own session logs show going wrong. Run Section 3.2, paste the output into Section 4, commit the receipt hashes, and the paper is done.