feat(tts): transcode WAV → Ogg/Opus + emit real waveform #91

Merged

jlxq0 merged 2 commits from feat/tts-transcode-to-ogg-opus into main

2026-05-22 15:17:14 +00:00

jlxq0 commented

2026-05-22 14:59:58 +00:00

Owner

Summary

Element only treats m.audio as a voice-memo bubble when the audio is Ogg/Opus — audio/wav falls back to the generic file-attachment UI regardless of the MSC3245 voice flag. So send_tts_voice_message was technically working but the bubble was dead in Element.

This PR transcodes Chatterbox's WAV output to Ogg/Opus in-process, in pure Rust:

New parse_wav_mono_16bit extracts PCM samples (mono / 16-bit / 8 · 12 · 16 · 24 · 48 kHz only, which covers Chatterbox).
New compute_waveform produces 40 peak-amplitude buckets in 0..=1024 — Element actually draws the bars now instead of a flat line.
New transcode_wav_to_ogg_opus runs the PCM through libopus (24 kbps Voip mode, 20 ms frames) and packs the packets into an RFC 7845 OggOpus container (OpusHead + OpusTags + audio pages, granule positions tracked at 48 kHz).
send_tts_voice_message_inner uploads the Ogg bytes as audio/ogg instead of audio/wav, includes the waveform in the MSC1767 audio block.

New crates: opus (libopus bindings) + ogg (container). opus builds libopus statically via cc-rs, so the distroless runtime image still needs no system libraries.

Test plan

cargo fmt
cargo clippy --all-targets -- -D warnings
cargo test (162 passed, +1 net: dropped 2 wav-duration tests, added 3 transcoder tests including a sine-wave roundtrip that asserts the output is a valid Ogg stream with ≥3 pages and a non-flat waveform)
After deploy: send a TTS voice message into a small test room, confirm Element X renders the voice-memo bubble with play button + waveform.

🤖 Generated with Claude Code

## Summary Element only treats `m.audio` as a voice-memo bubble when the audio is Ogg/Opus — `audio/wav` falls back to the generic file-attachment UI regardless of the MSC3245 voice flag. So `send_tts_voice_message` was technically working but the bubble was dead in Element. This PR transcodes Chatterbox's WAV output to Ogg/Opus in-process, in pure Rust: - New `parse_wav_mono_16bit` extracts PCM samples (mono / 16-bit / 8 · 12 · 16 · 24 · 48 kHz only, which covers Chatterbox). - New `compute_waveform` produces 40 peak-amplitude buckets in 0..=1024 — Element actually draws the bars now instead of a flat line. - New `transcode_wav_to_ogg_opus` runs the PCM through libopus (24 kbps Voip mode, 20 ms frames) and packs the packets into an RFC 7845 OggOpus container (OpusHead + OpusTags + audio pages, granule positions tracked at 48 kHz). - `send_tts_voice_message_inner` uploads the Ogg bytes as `audio/ogg` instead of `audio/wav`, includes the waveform in the MSC1767 audio block. New crates: `opus` (libopus bindings) + `ogg` (container). `opus` builds libopus statically via cc-rs, so the distroless runtime image still needs no system libraries. ## Test plan - [x] `cargo fmt` - [x] `cargo clippy --all-targets -- -D warnings` - [x] `cargo test` (162 passed, +1 net: dropped 2 wav-duration tests, added 3 transcoder tests including a sine-wave roundtrip that asserts the output is a valid Ogg stream with ≥3 pages and a non-flat waveform) - [ ] After deploy: send a TTS voice message into a small test room, confirm Element X renders the voice-memo bubble with play button + waveform. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jlxq0 added 1 commit

2026-05-22 14:59:59 +00:00

feat(tts): transcode WAV → Ogg/Opus + emit real waveform

CI / cargo (pull_request) Failing after 27s

Details

CI / docker (pull_request) Has been skipped

Details

faf7707938

Element only renders m.audio as a voice-memo bubble when the audio
is Ogg/Opus, regardless of the MSC3245 voice flag. WAV-mimetype
voice messages drop back to the generic file-attachment UI with no
play button or waveform bars, which is what users were seeing.

send_tts_voice_message now:

  - Takes Chatterbox's WAV bytes and decodes the mono 16-bit PCM
    samples in-process (no extra HTTP roundtrip, no ffmpeg).
  - Computes a 40-bucket peak waveform (0..=1024 per MSC1767) off
    those samples so Element draws real bars rather than a flat
    line.
  - Encodes the PCM through libopus at 24 kbps Voip mode and packs
    the resulting Opus packets into an RFC 7845 OggOpus container
    (OpusHead + OpusTags + audio pages, granule positions tracked
    at 48 kHz).
  - Uploads the Ogg bytes as audio/ogg.

Pulls two new crates (`opus`, `ogg`). `opus` builds libopus
statically from C source via opusic-sys/cc-rs, so the distroless
runtime image still needs no system libraries.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

jlxq0 added 1 commit

2026-05-22 15:12:10 +00:00

ci+docker: install cmake + force static libopus link

CI / cargo (pull_request) Successful in 2m14s

Details

CI / docker (pull_request) Successful in 2m22s

Details

67eeef66c2

The opus crate's build script (audiopus_sys) builds libopus from C
source via CMake. Two consequences in our CI / container build:

1. Both the Forgejo runner and the rust:1.93-bookworm builder image
   need cmake — without it, every cargo step (including clippy)
   fails inside build.rs at the cmake::build() call.

2. audiopus_sys's default linking on Linux/glibc is dynamic, which
   would make the distroless runtime image require libopus.so at
   start. Set OPUS_STATIC=1 in the workflow env and the Dockerfile
   so the static lib gets linked into the binary instead.

Both fixes are confined to build tooling; no runtime changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

jlxq0 merged commit 6f8fce1ea3 into main

2026-05-22 15:17:14 +00:00

jlxq0 deleted branch feat/tts-transcode-to-ogg-opus

2026-05-22 15:17:15 +00:00

jlxq0 referenced this pull request from a commit

2026-05-22 15:17:15 +00:00

feat(tts): transcode WAV → Ogg/Opus + emit real waveform (#91)