[carey-bk]

Summary

SentencepiecesTokenizer.ids2tokens interprets model output IDs using SentencePiece's internal ordering even when a different token_list was supplied. The published English Paraformer model has 10,020 model tokens and 10,000 BPE pieces with different ordering, producing unrelated words.

Delegate to BaseTokenizer.ids2tokens when a vocabulary exists, preserving pieces and word-boundary markers for tokens2text. Keep the existing decode fallback when no vocabulary was supplied. encode, decode, and the SenseVoice postprocessor are unchanged. Add 19 offline regression cases and wire them into the existing CPU workflow.

Related: #3780.

Validation

• Before the fix: 16 new regressions failed, 3 passed. After: all 19 passed, using real imported tokenizer and Paraformer methods.
• Real CPU inference with the public ModelScope checkpoint at 97ce0bc9b0178aac03008c3afe8271cdb534228f and its official 2.225-second English sample: procedure telegram impatience phuket sleeping danielot pro becomes he tried to think how it could be. All eight predicted IDs are identical before/after; all 956 loaded checkpoint entries were independently verified. No remote code was enabled.
• Existing CPU selection plus adjacent tests, on macOS arm64 / Python 3.12.14 / torch and torchaudio 2.10.0: 232 passed, 1 failed, 10 subtests passed, with both NumPy 1.26.4 and 2.4.6. The same unrelated ARM/librosa window-size failure reproduces on the unmodified base in both environments; no tests were skipped or weakened.
• Compilation, Black for the new test and changed method, dependency checks, workflow parsing, and diff checks passed. Two existing invalid-escape compilation warnings remain.
• Whole-file Black 24.4.0 checks now pass after normalizing existing docstring indentation in the touched tokenizer. A cache-independent stdin check was used. Final focused tokenizer/SenseVoice regression: 40 passed.

User impact and review notes

With a supplied vocabulary, ids2tokens now returns a piece list instead of decoded text. Callers requiring text should use tokens2text; without a vocabulary, legacy decoding remains unchanged. Please review this return-type correction and whether a migration note is needed.

Real-model evidence covers one unique official audio sample, downloaded from two matching public sources. No GPU, full accuracy benchmark, real SenseVoice checkpoint, or hosted CI validation was performed. Local NumPy 2 testing used 2.4.6 because 2.4.0 is yanked; the upstream CI matrix is unchanged.

Prepared with OpenAI Codex assistance.