Repository navigation
Score on the Voice Arena dictation benchmark; drop the llm_instruction length cap - #322
Merged
Merged
Conversation
…truction length cap
Voice Arena (voicearena.com/dictation-bench) ranks Blurt against thirteen other
Mac dictation tools. Its public split, 151 clips with recordings, is now part of
the dictation-prompt eval, and the 2048-character instruction cap is gone because
the API no longer enforces it.
- corpus/voicearena: `voicearena` source (VoiceArena/Dictation_Dataset, CC BY-NC).
Its input side is the service's own STT transcript of each recording (fetched
once, cached), since the STT already writes "$62,000" and "plan.startDate". Rows
are held out of train (dev/test only) and keep their long-form clips. Default
source is now nyra,voicearena; sources combine with commas.
- live: --verify-live plays recorded clips instead of `say` where a corpus has
them; --rewrite-mode smart sends rewrite_mode like the app's Smart setting.
- No length cap: re-probed 2026-11-18, llm_instruction of 2049-49152 chars is
accepted in Fast and Smart; only the 64 KiB config part is limited. Removed the
soft target, budgets, trimming and length gate; the proposer still gates
safeguards, field names and scaffolding. The Swift CleanupInstruction cap is
untouched; the CLI warns when a winner exceeds it.
- voicearena_baseline.py: scores Blurt on the public set with the leaderboard's
own judge (amend-va 0.2.1, Claude Opus 5.5) and attributes each failure to STT
or the rewrite. --instruction-file, --rewrite-mode and --host (EU zone).
- Cache Hugging Face row pages on disk: every run re-read ~40 nyra pages and hit
the rows API's rate limit, token or not.
- Pin DSPy below 3.5: 3.5.0 breaks every openai/ model call through the LLM
Gateway ("Not found"); 3.4.1 works.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`live` is already imported at module level; flagged by code scanning on #322. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dictation API no longer caps config.llm_instruction: the reference now documents it as `string | null` with no maxLength, and a probe on 2026-11-18 accepted 2049-49152 characters in Fast and Smart. Only the whole `config` part is limited (64 KiB). So a longer cleanup instruction can ship as-is. - CleanupInstruction: remove characterCap and customStyleBudget. sendable(appending:) always returns the instruction, with no over-cap fallback to the service default. - AssemblyAITranscriber: drop the unreachable over-cap guard and its log line. - StyleProfileStore.characterLimit: a fixed 4000-byte editor bound per profile, instead of the ~390 bytes of headroom the old cap left. The Mac and iPhone editors and the engine's trim all read it. Only the active profile is still sent, now justified on its own terms rather than by the cap. - Tests pin the new behaviour: the instruction is always sent, a style is trimmed only to the editor limit (by bytes, on whole characters), and the longest request stays far inside the config limit. - AGENTS.md, the engine README and the eval's notes describe the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a run A 4-hour Smart search crashed at its last step. The Voice Arena audio URLs are signed and had expired by the time --verify-live played them (403), and the error escaped, so --out was never written. - voicearena.ensure_audio: download the 151 recordings once at load, while the URLs are fresh (~100 MB, kept beside the STT cache). Utterances point at the local files. The STT step saves what it downloads too. - live: play local recordings without the network. Refresh any URL that is still a URL just before use, and tolerate the datasets-server failing there. - run_live_verification: catch any failure, report it and carry on. The search's result outranks the live check. - describe_length: no cap warning, since the app's cap is gone too. - voicearena_baseline: play the local copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clip ids come from a third-party dataset and name the cached recordings, so an id like ../../x could have written outside the cache. Keep [A-Za-z0-9_-] only and check the path stays in the directory. Real ids (dict_0007) are unchanged, so existing cached files still match. Flagged by automated security review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
502s and 500s failed three loads in one afternoon, each fine on a manual rerun seconds later, and each threw a run away before its first model call. Retry 500/502/503/504 with the same logged backoff as a rate limit; a 4xx still fails at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two suites that landed on main after this branch began still treated the cleanup instruction as possibly nil (a redundant #require, a dead ?? ""), which CI's warnings-as-errors build rejects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Smart search tied the shipped prompt on Voice Arena. Every row GEPA ranked on and learned from was Switchboard conversation, so the dictation failures that separate prompts there never reached it. - --voicearena-in-search [SHARE] (0.5 when bare) releases that share of the held-out Voice Arena rows into the search. carve_validation splits them evenly between GEPA's valset and its reflection set. The rest stay in dev/test to catch overfitting. Off by default; nothing changes without it. - --seed-file PATH starts the search from a prompt in a file (e.g. the last run's winner). It is gated like any proposal and scored on dev next to BASELINE. - The split printout lists the valset on its own line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rewrite runs a different model per mode, and one instruction no longer fits
both. CleanupInstruction now has two:
- `text`, Fast's, unchanged.
- `smartText`, Smart's: `text` with two edits, derived with `replacing` so the
difference is explicit. It keeps greetings and meaningful framing ("Hey,",
"just letting you know") that the aggressive opener rule was deleting, and
deletes "kind of" only as filler.
Scored on the Voice Arena public set (151 clips) with the benchmark's own judge,
two independent samples each:
- Smart: ZER 90.1 / 88.1 vs 87.4 / 87.4, AMEND 0.16 / 0.20 vs 0.33 / 0.26.
No clip the old instruction passed in both samples failed in both.
- Fast: ZER 75.5 / 76.2 vs 78.1 / 78.1, so Fast keeps `text`.
The transcriber reads the mode once per request and uses it for both the
instruction and the echo check. With enhanced transcripts off no rewrite_mode is
sent, so the service runs Fast and Fast's instruction goes with it.
Eval: candidates.SMART_WINNER is derived the same way, and a test applies the
Swift edit constants to PRIOR_WINNER to keep the two byte-identical.
--baseline smart-winner makes a Smart search's bar Smart's own instruction.
voicearena_baseline gains --sample for independent repeat runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nner StreamingPCMPlayerTests' drain bound restated drainSlack and allowed 0.5 s for the poll loop and the scheduler. On CI, with no output device, the drain waits out its whole deadline, and a loaded runner took 2.85 s against the 2.75 s bound. Read drainSlack itself (now internal) and allow 2 s. The test guards a drain that never ends, which no margin of seconds lets through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the public split of the Voice Arena dictation benchmark into the dictation-prompt eval, and removes the 2048-character
llm_instructioncap from both the prompt search and the app, now that the API no longer enforces it.Voice Arena in the eval
voicearenasource (VoiceArena/Dictation_Dataset, 151 clips with recordings, CC BY-NC 4.0, loaded at runtime and never committed).voicearena.py). The STT already writes "$62,000" and "plan.startDate", sospoken_textwould make the search spend itself on number and code formatting the rewrite never sees.nyra,voicearena, and sources combine with commas.--verify-liveplays recorded clips instead ofsaywhere a corpus has them.--rewrite-mode smartsendsrewrite_modethe way the app's Smart setting does.voicearena_baseline.pyscores Blurt on the public set with the benchmark's own judge (amend-va0.2.1). It attributes each failing clip to STT or to the rewrite, and has--instruction-file,--rewrite-modeand--host(EU zone).No length cap
configpart is limited (64 KiB).1fe9169).CleanupInstructionhas nocharacterCap, andsendable(appending:)always returns the instruction, so a longer prompt ships as-is. The transcriber's over-cap fallback is gone.StyleProfileStore.characterLimit), instead of the ~390 bytes of headroom the old cap left. It's still far inside the service's 64 KiBconfiglimit.Smart gets its own cleanup instruction
CleanupInstruction.smartTextistextplus two edits, derived withreplacing: it keeps greetings and meaningful framing ("Hey,", "just letting you know"), which the aggressive opener rule was deleting, and deletes "kind of" only as filler. Fast keepstextunchanged.Reliability fixes
openai/model call through the LLM Gateway with "Not found"; 3.4.1 works.--verify-livegot a 403.--verify-livecan no longer crash a finished run. Any failure is reported and the run carries on, so--outis always written.Test plan
scripts/check.sh(full: Swift build, all test suites, SwiftLint, periphery, iOS checks, invariants)scripts/check.sh --portable: ruff format and check, pytest (152 tests, including new ones for the held-out split, multi-source loading, the STT cache, recorded-audio playback,rewrite_mode, page caching and the gate-only proposer), prettier and markdownlint--dry-runon the realnyra,voicearenacorpus--optimizer nonethrough the LLM Gatewayvoicearena_baseline.pyfull runs on the public set (US Smart, US Fast, EU Smart)🤖 Generated with Claude Code