Makes ../../voice/: one MP3 per couplet (voice/N/K.mp3: line 1, a pause, line 2) and voice/index.js, which tells
js/voice.js where each clip sits on its scene’s clock and when each character is spoken. The page only plays these
files; nothing here runs in the browser.
tools/voice/build.sh # everything that is missing or out of date
tools/voice/build.sh --poems 12,290-300 # only these poems
It is incremental. A couplet is recorded again only when its text, its reading (overrides.json) or the voice
(voice.json) changed; pack then re-cuts only those clips and rewrites index.js. After adding poems, run it again.
node, ffmpeg (with libmp3lame), an NVIDIA GPU.Two Python environments, because the speech model and the checker pin different transformers versions.
build.sh looks for them in $VOICE_ENV (default <repo>/.claude/scratch/voice):
cd <repo>/.claude/scratch/voice
uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python torch torchaudio --index-url https://download.pytorch.org/whl/cu130
uv pip install --python .venv/bin/python qwen-tts pypinyin opencc-python-reimplemented soundfile numpy
uv venv --python 3.12 .venv-asr && uv pip install --python .venv-asr/bin/python torch torchaudio --index-url https://download.pytorch.org/whl/cu130
uv pip install --python .venv-asr/bin/python qwen-asr pypinyin opencc-python-reimplemented soundfile numpy
plan.json, report.json) go to $VOICE_WORK (default $VOICE_ENV/work). They
are a cache: deleting them only means recording again.voice.py <step>)| step | what it does |
|---|---|
lines.mjs |
reads the poem data the way prepare() in js/player.js does → every couplet’s lines, punctuation, cps, dur |
plan |
the reading of every character, and the text sent to the model |
synth |
records the couplets that have no accepted recording (Qwen3-TTS, a fixed preset voice, 16 couplets at a time) |
check |
transcribes each recording and aligns it to the text; accepts or rejects it |
pack |
cuts at the pause between the lines, places the lines on the scene’s clock, evens out loudness, encodes, writes index.js |
status |
counts, failures, and the readings that could not be forced → report.json |
review |
prints every line with its reading and its English, for proof-reading |
sounds |
rebuilds sounds.json (needs wordfreq; only when pypinyin is upgraded) |
A rejected recording is made again, up to tries times: tries 0 and 1 with the planned text, later tries with the
characters the transcript got wrong replaced by stand-ins (below). build.sh runs synth → check four times.
The model is given characters, not pinyin, so a polyphonic character is made unambiguous before it is sent:
plan drafts each line’s reading with pypinyin on the simplified text. A character inside a word pypinyin knows
keeps that word’s reading; a lone character takes its usual reading in verse from verse.json (長 cháng, 還 huán,
了 liǎo …). 一 and 不 keep their citation tones (the model applies the sandhi).overrides.json corrects the draft: {"290.3.1": {"爲": "wei2"}} = poem 290, couplet 3, line 1. A key "重@5"
means the fifth character only. Readings are pinyin with a tone digit, ü written v.
The standard is modern Mandarin as dictionaries give it for the sense in context; no rhyme-driven old readings.sounds.json → dom). Otherwise it is replaced by a common character that has only that sound
(sounds.json → homo): 還 read huán is sent as 环, 朝 read zhāo as 招. Characters with no clear default
(爲, 長, 重, 行 …) are always replaced, and so is any character outside GB2312, which a model may never have met
(for those sounds.json → rare supplies a less common stand-in when homo has none). The listener hears the
same syllable; only the model sees the stand-in.report.json under unforced.{"443.7": {"say": "晡糟醉方熟,洒面呼不醒。"}}. The readings above are still what
the recording is checked against.What check can and cannot tell: it catches missing, extra and wrong syllables, runaway recordings, lines run
together. It cannot hear tones, and a transcript does not show which reading of a polyphonic character was spoken
(it writes 還 for both), so the readings rest on steps 1–3, and on ears.
check hears a recording twice when needed. The first hearing is blind (the transcriber is not told the text) and
often writes a likelier modern word for a correct sound. A recording rejected only for one or two such syllables,
with the right number of syllables, is heard again with the text as a hint; if the hinted transcript agrees, it is
accepted. Anything with a syllable too many or too few is recorded again instead. If no try passes, the nearest miss
(at most two syllables heard differently) is used and listed in report.json under doubt; listen to those.
The scene’s uniform beats are l1 = 0.5, l2 = l1 + n1·cps + 0.3 (beatTimes() in js/stage.js). pack puts the
first sound of line 1 at l1 + 0.12 and of line 2 at l2 + 0.12, or later if line 1 has not finished 0.28 s before
that. Every clip opens with 0.10 s of silence. index.js holds, per couplet,
[id, t0, d, e2, [line-1 character times], [line-2 character times]] in hundredths of a second: the scene time at
which the clip starts, its length, the clip time at which line 2 starts, and the scene time of every character.
Edit voice.json (speaker: one of the model’s preset voices; instruct: how to read) and run build.sh: every
couplet is recorded again. To try a voice without touching the site:
VOICE_WORK=/some/dir tools/voice/build.sh --config my-voice.json --out /some/dir/voice --poems 94,95