The reading voice

The reading voice

Makes ../../voice/: one MP3 per couplet (voice/N/K.mp3: line 1, a pause, line 2) and voice/index.js, which tells js/voice.js where each clip sits on its scene’s clock and when each character is spoken. The page only plays these files; nothing here runs in the browser.

tools/voice/build.sh                     # everything that is missing or out of date
tools/voice/build.sh --poems 12,290-300  # only these poems

It is incremental. A couplet is recorded again only when its text, its reading (overrides.json) or the voice (voice.json) changed; pack then re-cuts only those clips and rewrites index.js. After adding poems, run it again.

What it needs

  • node, ffmpeg (with libmp3lame), an NVIDIA GPU.
  • Two Python environments, because the speech model and the checker pin different transformers versions. build.sh looks for them in $VOICE_ENV (default <repo>/.claude/scratch/voice):

    cd <repo>/.claude/scratch/voice
    uv venv --python 3.12 .venv     && uv pip install --python .venv/bin/python     torch torchaudio --index-url https://download.pytorch.org/whl/cu130
    uv pip install --python .venv/bin/python     qwen-tts pypinyin opencc-python-reimplemented soundfile numpy
    uv venv --python 3.12 .venv-asr && uv pip install --python .venv-asr/bin/python torch torchaudio --index-url https://download.pytorch.org/whl/cu130
    uv pip install --python .venv-asr/bin/python qwen-asr pypinyin opencc-python-reimplemented soundfile numpy
    
  • Work files (recordings, verdicts, plan.json, report.json) go to $VOICE_WORK (default $VOICE_ENV/work). They are a cache: deleting them only means recording again.

The steps (voice.py <step>)

step what it does
lines.mjs reads the poem data the way prepare() in js/player.js does → every couplet’s lines, punctuation, cps, dur
plan the reading of every character, and the text sent to the model
synth records the couplets that have no accepted recording (Qwen3-TTS, a fixed preset voice, 16 couplets at a time)
check transcribes each recording and aligns it to the text; accepts or rejects it
pack cuts at the pause between the lines, places the lines on the scene’s clock, evens out loudness, encodes, writes index.js
status counts, failures, and the readings that could not be forced → report.json
review prints every line with its reading and its English, for proof-reading
sounds rebuilds sounds.json (needs wordfreq; only when pypinyin is upgraded)

A rejected recording is made again, up to tries times: tries 0 and 1 with the planned text, later tries with the characters the transcript got wrong replaced by stand-ins (below). build.sh runs synth → check four times.

Readings

The model is given characters, not pinyin, so a polyphonic character is made unambiguous before it is sent:

  1. plan drafts each line’s reading with pypinyin on the simplified text. A character inside a word pypinyin knows keeps that word’s reading; a lone character takes its usual reading in verse from verse.json (長 cháng, 還 huán, 了 liǎo …). 一 and 不 keep their citation tones (the model applies the sandhi).
  2. overrides.json corrects the draft: {"290.3.1": {"爲": "wei2"}} = poem 290, couplet 3, line 1. A key "重@5" means the fifth character only. Readings are pinyin with a tone digit, ü written v. The standard is modern Mandarin as dictionaries give it for the sense in context; no rhyme-driven old readings.
  3. A character is sent as itself when the wanted reading is the one a modern speaker gives it by default (sounds.json → dom). Otherwise it is replaced by a common character that has only that sound (sounds.json → homo): 還 read huán is sent as 环, 朝 read zhāo as 招. Characters with no clear default (爲, 長, 重, 行 …) are always replaced, and so is any character outside GB2312, which a model may never have met (for those sounds.json → rare supplies a less common stand-in when homo has none). The listener hears the same syllable; only the model sees the stand-in.
  4. Where no stand-in exists (更 gèng, 樂 lè, 了 liǎo …) the character is sent as it is; these are listed in report.json under unforced.
  5. When the model still will not say a couplet right, give the text to send outright (simplified, one character per syllable, with the punctuation): {"443.7": {"say": "晡糟醉方熟,洒面呼不醒。"}}. The readings above are still what the recording is checked against.

What check can and cannot tell: it catches missing, extra and wrong syllables, runaway recordings, lines run together. It cannot hear tones, and a transcript does not show which reading of a polyphonic character was spoken (it writes 還 for both), so the readings rest on steps 1–3, and on ears.

check hears a recording twice when needed. The first hearing is blind (the transcriber is not told the text) and often writes a likelier modern word for a correct sound. A recording rejected only for one or two such syllables, with the right number of syllables, is heard again with the text as a hint; if the hinted transcript agrees, it is accepted. Anything with a syllable too many or too few is recorded again instead. If no try passes, the nearest miss (at most two syllables heard differently) is used and listed in report.json under doubt; listen to those.

Timing

The scene’s uniform beats are l1 = 0.5, l2 = l1 + n1·cps + 0.3 (beatTimes() in js/stage.js). pack puts the first sound of line 1 at l1 + 0.12 and of line 2 at l2 + 0.12, or later if line 1 has not finished 0.28 s before that. Every clip opens with 0.10 s of silence. index.js holds, per couplet, [id, t0, d, e2, [line-1 character times], [line-2 character times]] in hundredths of a second: the scene time at which the clip starts, its length, the clip time at which line 2 starts, and the scene time of every character.

Changing the voice

Edit voice.json (speaker: one of the model’s preset voices; instruct: how to read) and run build.sh: every couplet is recorded again. To try a voice without touching the site:

VOICE_WORK=/some/dir tools/voice/build.sh --config my-voice.json --out /some/dir/voice --poems 94,95