Audio codecs for the speech models that gbx-lm serves, on Apple silicon and NVIDIA (DGX Spark).
A speech model generated by gbx-lm produces codes, not audio. This package turns codes into audio, and reference audio into codes. It runs as a separate process that gbx-lm starts and talks to over a pipe, so the server itself needs none of these dependencies, and the codec can be installed, upgraded or removed without touching gbx-lm.
| codec | used by | rate | codebooks |
|---|---|---|---|
fish_s1 (Fish Audio S1 DAC) |
Fish Audio Drama 3 | 44.1 kHz mono, 21.5 frames/s | 10 |
The codec weights come with the model repository; this package holds code only.
Python 3.10 or newer, in a virtual environment of its own:
python3 -m venv ~/.gbx-codecs
~/.gbx-codecs/bin/pip install gbx-codecs # Apple silicon Mac
~/.gbx-codecs/bin/pip install "mlx[cuda13]" gbx-codecs # Linux with an NVIDIA GPU (DGX Spark)Dependencies: mlx and numpy. No PyTorch and no ffmpeg. Reference audio is read from WAV; for flac or mp3, add gbx-codecs[formats] (which installs miniaudio; on Linux ARM it compiles, and needs the Python headers). The process needs about 3 GB of memory while decoding (Fish S1: 1.75 GB of weights, decoded 8 frames at a time).
Check it against a downloaded model:
~/.gbx-codecs/bin/gbx-codecs check /path/to/the/model
# codec OK: 10 codebooks, 44 frames for 2 s, 44100 Hz, round-trip correlation 0.9..Then tell gbx-lm which Python to start it with (--codec_python ~/.gbx-codecs/bin/python, or GBX__SERVER__CODEC_PYTHON). See the model card of the speech model for the full setup.
gbx-codecs encode MODEL_DIR reference.wav codes.npy # audio -> int32 codes [codebooks, frames]
gbx-codecs decode MODEL_DIR codes.npy out.wav # codes -> 16-bit wav
gbx-codecs serve # the companion process gbx-lm startsgbx-codecs serve reads requests on stdin and writes replies on stdout; logs go to stderr. Each message is GBXC, a 4-byte header length and an 8-byte payload length (big-endian), a UTF-8 JSON header, then the payload. Operations: hello, load (codec, model directory), encode (audio file to int32 codes), begin (output sample rate), decode (int32 codes to 16-bit PCM, streamed), end, quit. See src/gbx_codecs/protocol.py and src/gbx_codecs/worker.py.
Decoding is streaming: frames sent a piece at a time decode to the same audio whatever the size of the pieces, to within one 16-bit step, and resampling (for example to 24 kHz for OpenAI-style pcm output) carries its state across pieces.
MIT, see LICENSE.
The fish_s1 codec implements Fish Audio's audio codec, the one Fish Audio Drama 3 uses. It was ported from mlx-audio; use it with Fish Audio's codec weights only as the Fish Audio Community License that comes with those weights allows.
The Fish Audio S1 DAC implementation in src/gbx_codecs/fish_s1/dac.py is ported from mlx-audio (MIT License, Copyright (c) 2024 Prince Canuma; see licenses/LICENSE.mlx-audio.txt). The codec weights are distributed with the models under the model's own license; this package does not include them.
Not affiliated with or endorsed by Fish Audio.