Skip to content

Voice: make the transcription language configurable (currently hard-coded to "auto") #5226

Open

Description

@yaccob

Describe the feature or problem you'd like to solve

Voice dictation (/voice) works with the multilingual Nemotron streaming model, but the voice worker always starts
the live transcription session with automatic language detection. For non-English speakers this means the first
sentence of every dictation is recognised noticeably worse.

Environment: Copilot CLI 1.0.101 (the same code is in 1.0.96–1.0.100), foundry-local-sdk 2.1.0, model
nemotron-3.5-asr-streaming-0.6b-generic-cpu:3, macOS arm64 (Apple Silicon).

voice-engine.worker.js → openStreamingSession() sets the language before starting the session (minified):

let r = t.createAudioClient().createLiveTranscriptionSession();
r.settings.language = "auto";
try { await r.start() } catch (i) { … }

The model is prompt-conditioned on a language id (lang_id). With "auto" it commits to a language only at the end
of the first sentence (it emits a <de-DE> tag there), so the first sentence is decoded without the language prior.

Measured with a small Node script that makes the same SDK calls as the worker (createLiveTranscriptionSession,
settings.language, start, append in real-time-paced 100 ms chunks of 16 kHz mono s16le) on German TTS samples,
each with 1 s of leading silence so that the whole first sentence reaches the model. Reference: "Bitte prüfe, warum
der Build in der Pipeline fehlschlägt. Schau dir dazu die Logs im Kubernetes-Cluster an …"
. First sentence of the
result:

settings.language First sentence
"auto" Bitte prüfe, warum der Bult in der Pipeline fehlt.
"de-DE" or "de" Bitte prüfe, warum der Bult in der Pipeline fehlschlägt.
"en-US" German speech is dropped
unset or numeric no text at all

The difference is reproducible; the rest of the text is identical once the model has detected the language.

Proposed solution

Make the language configurable: a setting (e.g. voice.language, default "auto") and/or /voice language de-DE,
passed to settings.language. Validate the value against the tags the model supports, since invalid values silently
produce no text and a wrong language drops speech.

Additional context

  • Related: multilingual voice input #3658 (multilingual voice input). This request is narrower: the bundled model already handles other
    languages; only the language hint is hard-coded.
  • A user-level workaround exists: patching the minified worker in the CLI package cache, replacing
    X.settings.language="auto" with X.settings.language="de-DE". It works, but has to be re-applied after every CLI
    update.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions