fix(tts): speak the tilde in digit ranges instead of mashing the numbers
"20~30초" is read aloud as a single number — OmniVoice says "이십삼" (23).
The separator never reaches the listener, so any written range is heard as
the wrong figure.
`normalize_text` only ran its number pass behind `_num2words_lang`, which
returns None for ko/ja/zh/th/vi (those scripts read digits natively and are
deliberately outside num2words). Nothing else looked at the range mark, so
the tilde went to the engine untouched and the two numbers ran together.
Rewrite `N~M` into the spoken form before the engine sees it, outside the
num2words gate so the CJK languages are covered too. Verified by rendering
each candidate and transcribing it back (ko, OmniVoice, cloned voice):
"대략 20~30초짜리" heard "23초" WRONG
"대략 20-30초짜리" heard "23초" WRONG (reproduces it)
"대략 20에서 30초짜리" heard "20에서 30초짜리" correct
"20〜30分ぐらい" → "20から30分" heard "20〜30分くらい" correct
Deliberately narrow:
* Only the tilde family (U+007E, U+301C, U+FF5E). Japanese and Korean IMEs
emit the latter two. An ASCII hyphen is left alone — between digits it
also spells dates, phone numbers and product codes, where "to" is wrong
(`tests` already pin "pages 3-5" as unchanged).
* Only languages with a verified spoken form (ko/ja/zh/en). Anything else
keeps its tilde, matching how `_PERCENT_WORD` is scoped.
* Spacing belongs to the form, not the caller: a Korean postposition binds
to its numeral ("20에서 30"), Japanese and Chinese set no spaces, English
needs them on both sides.
* Neighbour guards block digits and ASCII letters but allow CJK, because
CJK writes the unit hard against the digits ("20~30초"); a `\w` guard
rejects exactly the cases the rule exists for.
`ko`/`ja`/`zh` join `_FULL_NAME_TO_CODE` so the new resolver can see them.
They stay out of `_NUM2WORDS_LANGS`, so this does not open a num2words path
for them — the same inert-entry pattern the file already documents for
"vietnamese".
`backend/services/text_normalization.py` joins the functional-CJK allowlist
in tests/test_no_hardcoded_cjk.py, under the text-processing group and by
the procedure that file documents: the range words are engine input, not
user-facing UI strings.
Tests: 7 new change-cases and 8 new leave-unchanged cases (hyphen, date,
phone number, product code, decimals, a non-numeric tilde, an unverified
language, and no language at all). All 7 change-cases fail against the
previous implementation.
Full suites before and after: the same 17 failures, none of them touched by
this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
f2302e8c95
commit
ed746bab57
@@ -10,6 +10,8 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
|
||||
**Highlights**
|
||||
|
||||
- Number ranges are read as ranges — "20~30초" no longer comes out as "23" (#1821)
|
||||
|
||||
### Changed
|
||||
|
||||
### Added
|
||||
@@ -18,6 +20,8 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Digit ranges written with a tilde are now spoken: `20~30초` read aloud as "23" because the separator never reached the engine, and the two numbers ran together (#1821)
|
||||
|
||||
|
||||
## [0.5.2] — 2026-09-02
|
||||
|
||||
|
||||
@@ -112,6 +112,13 @@ _FULL_NAME_TO_CODE = {
|
||||
"vietnamese": "vi",
|
||||
"kazakh": "kz",
|
||||
"standard arabic": "ar",
|
||||
# Below: inert for num2words (absent from _NUM2WORDS_LANGS, which reads
|
||||
# digits natively for these scripts), present so _plain_lang_code can
|
||||
# resolve them for the digit-range rule.
|
||||
"korean": "ko",
|
||||
"japanese": "ja",
|
||||
"chinese": "zh",
|
||||
"mandarin chinese": "zh",
|
||||
}
|
||||
|
||||
# ISO codes whose num2words locale name differs.
|
||||
@@ -178,6 +185,68 @@ def _num2words_lang(language: Optional[str]) -> Optional[str]:
|
||||
return None
|
||||
|
||||
|
||||
def _plain_lang_code(language: Optional[str]) -> Optional[str]:
|
||||
"""Resolve a request language to a bare ISO code, with no num2words gate.
|
||||
|
||||
:func:`_num2words_lang` answers "may I call num2words for this?" and so
|
||||
returns ``None`` for ko/ja/zh/th/vi. Rules that are not num2words-backed
|
||||
need the code itself, which is what this returns.
|
||||
"""
|
||||
if not language:
|
||||
return None
|
||||
s = str(language).strip().lower()
|
||||
if not s or s == "auto":
|
||||
return None
|
||||
code = _FULL_NAME_TO_CODE.get(s)
|
||||
if code:
|
||||
return code
|
||||
m = _ISO_CODE_RE.match(s)
|
||||
if m:
|
||||
return _ISO_ALIASES.get(m.group(1), m.group(1))
|
||||
return None
|
||||
|
||||
|
||||
# ── Digit ranges ─────────────────────────────────────────────────────────────
|
||||
# "20~30" loses its separator at the engine and reads as ONE number: OmniVoice
|
||||
# says "이십삼" (23) for "20~30초". Speak the separator instead. Verified by
|
||||
# rendering each form and transcribing it back (ko, OmniVoice):
|
||||
# "20~30초" → heard "23초" ✗
|
||||
# "20에서 30초" → heard "20에서 30초" ✓
|
||||
# Only the tilde family is rewritten — those are unambiguously range marks
|
||||
# between digits. An ASCII hyphen is left alone on purpose: it also spells
|
||||
# dates, phone numbers and product codes, where "to" would be wrong.
|
||||
#: Spacing is part of the form, not decoration: a Korean postposition binds to
|
||||
#: the numeral ("20에서 30"), Japanese and Chinese set no spaces at all, and
|
||||
#: English needs them on both sides.
|
||||
_RANGE_FORM = {
|
||||
"ko": "{a}에서 {b}",
|
||||
"ja": "{a}から{b}",
|
||||
"zh": "{a}到{b}",
|
||||
"en": "{a} to {b}",
|
||||
}
|
||||
|
||||
#: ASCII tilde, wave dash, fullwidth tilde — Japanese and Korean IMEs emit the
|
||||
#: latter two, so all three have to match.
|
||||
#:
|
||||
#: The neighbour guards block digits/decimal marks (so a longer number is never
|
||||
#: split) and ASCII letters (so a product code like "AB20~30CD" is left alone),
|
||||
#: but deliberately allow everything else: CJK writes its unit right against
|
||||
#: the digits — "20~30초", "20〜30分", "20~30秒" — and a \w guard would reject
|
||||
#: exactly the cases this rule exists for.
|
||||
_NUM_RANGE_RE = re.compile(
|
||||
r"(?<![\d.,])(?<![A-Za-z])(\d{1,6})\s*[~\u301c\uff5e]\s*(\d{1,6})"
|
||||
r"(?![\d.,])(?![A-Za-z])"
|
||||
)
|
||||
|
||||
|
||||
def _speak_number_ranges(text: str, lang: str) -> str:
|
||||
"""``20~30`` → ``20에서 30``. No-op where the spoken form isn't verified."""
|
||||
form = _RANGE_FORM.get(lang)
|
||||
if not form:
|
||||
return text
|
||||
return _NUM_RANGE_RE.sub(lambda m: form.format(a=m.group(1), b=m.group(2)), text)
|
||||
|
||||
|
||||
# ── Universal safety filters (all languages) ─────────────────────────────────
|
||||
|
||||
# Zero-width & bidi controls, C0/C1 controls (except \t \n \r), BOM, U+FFFD.
|
||||
@@ -487,6 +556,11 @@ def normalize_text(text: str, language: Optional[str] = None) -> str:
|
||||
if not text:
|
||||
return text or ""
|
||||
out = _safety_filters(text)
|
||||
# Runs outside the num2words gate below: ko/ja/zh keep their digits (that
|
||||
# gate returns None for them) but still need the range mark spoken.
|
||||
plain = _plain_lang_code(language)
|
||||
if plain:
|
||||
out = _outside_brackets(out, lambda t: _speak_number_ranges(t, plain))
|
||||
lang = _num2words_lang(language)
|
||||
if lang:
|
||||
if lang in _ABBREV_COMPILED:
|
||||
|
||||
@@ -57,6 +57,7 @@ _ALLOWED_FILES = {
|
||||
"backend/services/segmentation.py",
|
||||
"backend/services/sentence_chunker.py", # streaming-TTS terminator tables (Patter port, Wave 1.4)
|
||||
"backend/services/subtitle_segmenter.py",
|
||||
"backend/services/text_normalization.py", # spoken range word per language ("20~30" must not read as one number)
|
||||
"backend/core/http_headers.py", # docstring quotes the CJK filename that 500'd the header (#1262)
|
||||
"frontend/src/components/DubSegmentRow.jsx",
|
||||
"frontend/src/components/StoriesEditor.jsx",
|
||||
|
||||
@@ -65,6 +65,18 @@ _CHANGE_CASES = [
|
||||
("French", "il a 42 chats", "il a quarante-deux chats"),
|
||||
("French", "Mme Dupont arrive", "Madame Dupont arrive"),
|
||||
("Russian", "у меня 42 кота", "у меня сорок два кота"),
|
||||
# Digit ranges: the tilde has to be SPOKEN or the engine mashes the two
|
||||
# numbers into one ("20~30초" was read as "이십삼"). Spacing is part of the
|
||||
# per-language form — a Korean postposition binds to its numeral, Japanese
|
||||
# and Chinese set no spaces, English needs them.
|
||||
("Korean", "대략 20~30초짜리", "대략 20에서 30초짜리"),
|
||||
("Korean", "가격은 20~30만원", "가격은 20에서 30만원"),
|
||||
("ko", "20~30초", "20에서 30초"),
|
||||
("Japanese", "20〜30分ぐらい", "20から30分ぐらい"), # wave dash U+301C
|
||||
("Japanese", "20~30分ぐらい", "20から30分ぐらい"), # fullwidth U+FF5E
|
||||
("Chinese", "大约需要20~30秒", "大约需要20到30秒"),
|
||||
# EN runs the range through num2words afterwards, as it does any digit
|
||||
("English", "It takes 20~30 seconds", "It takes twenty to thirty seconds"),
|
||||
# Universal safety filters (language-independent)
|
||||
(None, "hello world", "hello world"),
|
||||
(None, "too many\t spaces", "too many spaces"),
|
||||
@@ -124,6 +136,16 @@ _UNCHANGED_CASES = [
|
||||
("English", "I said no. Fine."), # the word "no.", not "number"
|
||||
("English", "down main st. Anyway"), # lowercase "st." is not Saint
|
||||
("German", "es kostet 3,5 Euro"), # decimal comma: ambiguous
|
||||
# Digit ranges: only the tilde family is a range mark. Everything else that
|
||||
# sits between digits means something other than "to".
|
||||
("Korean", "대략 20-30초"), # ASCII hyphen: also dates/phones
|
||||
("Korean", "2026-09-05 회의"), # date
|
||||
("Korean", "010-1234-5678"), # phone number
|
||||
("Korean", "AB20~30CD"), # product code, not a range
|
||||
("Korean", "1.20~30.5"), # decimals either side
|
||||
("Japanese", "そうですね〜"), # tilde not between digits
|
||||
("Vietnamese", "20~30 giây"), # no verified spoken form
|
||||
(None, "20~30초"), # no language given
|
||||
# Unsupported languages keep every digit (num2words unmapped)
|
||||
("Japanese", "42 cats and 3.5 stars at 3:30"),
|
||||
("Thai", "42 cats"),
|
||||
|
||||
Reference in New Issue
Block a user