Repository navigation
Whisper adapter - #1944
Merged
Merged
Whisper adapter#1944
Conversation
Audio encoder + text decoder models (Whisper) fit neither existing audio path: encoder-only audio models drop the tokenizer and feed `input` as input_values, and audio-text LMs load as AutoModelForSeq2SeqLM. - New AUDIO_SEQ2SEQ_ARCHITECTURES set, routed to AutoModelForSpeechSeq2Seq and classified as seq2seq. - Explicit cfg.is_audio_seq2seq flag. skip_tokenizer_for_modality keeps the tokenizer for it, and the processor loader runs AutoProcessor after the audio feature extractor so bridge.processor ends as the full processor (feature extractor + tokenizer). Audio-text LMs are unaffected. - forward(): the adapter declares the keyword audio goes in under (ArchitectureAdapter.audio_input_name); audio seq2seq models take their features as `input`, require decoder_input (or labels), and skip the shift-from-input_ids that text encoder-decoders use. 1-D decoder ids are batched. The input_values branch is untouched. - enable_compatibility_mode() refuses audio seq2seq with an accurate message until encoder weight processing lands. - n_ctx falls back to max_target_positions before the 2048 default.
So verify_models can run on Whisper: - build_modality_input builds the [batch, n_mels, 2 * n_frames] log-mel window for audio seq2seq. - The Phase 1 HF reference capture and benchmark_forward_pass pass the features under the adapter's audio_input_name plus decoder_start decoder ids, the same single-token convention text encoder-decoders use. - The component benchmark skips the channel-first front-end convs (covered end to end by the forward pass), and decoder_embed always gets token ids instead of the modality input. - The encoder-only Phase 8 audio suite skips audio seq2seq.
WhisperForConditionalGeneration (tiny through large-v3, large-v3-turbo,
distil-whisper) loads through boot_transformers with hooks in both the
audio encoder and the text decoder:
logits, cache = bridge.run_with_cache(input_features, decoder_input=ids)
WhisperArchitectureAdapter subclasses BartFamilyArchitectureAdapter
(pre-LN, per-stack final norms). What differs from the BART base:
- Encoder front end: conv1 / conv2 (channel-first, so their hooks declare
position on dim 2; hook_out is pre-GELU) and the fixed sinusoidal
embed_positions table, read whole as [n_frames, d_model], in place of
token embeddings. Two small seams on the base make this declarative:
_encoder_embeddings() and unembed_name.
- Unembedding is proj_out, tied to decoder_embed, with no
final_logits_bias.
- Stacks may be asymmetric (large-v3-turbo 32/4, distil-whisper n/2);
n_layers follows the decoder, as for Blenderbot.
- k_proj has no bias while q/v/out_proj do; noted in the docstring and
pinned by the tiny and whisper-tiny tests.
Out of scope here and planned as follow-ups: generate()
(supports_generation=False) and weight processing (compatibility mode
refuses). applicable_phases=[1] follows from that.
Registered in all four places. The model_type fallback and 17 registry
entries are added at status 0 (openai/whisper-*, distil-whisper/*).
Tests:
- unit: mapping, flags, routing, tokenizer carve-out, n_ctx, factory.
- test_whisper_tiny.py (CI, random weights, no download): independent HF
parity for logits and labels-only loss, hook shapes on both stacks,
front-end math, pos_slice on the declared axes, an encoder-output patch
that equals HF fed the patched encoder_outputs, error paths, and the
Phase 1 benchmarks.
- test_whisper_adapter.py (slow, openai/whisper-tiny): parity against an
independent eager HF model, the k_proj bias, and hook shapes.
Refs TransformerLensOrg#1938
verify_models on openai/whisper-tiny: P1 at 100% (all 74 components equivalent to HF, forward-pass logits within tolerance). P2-P4 are not applicable yet (applicable_phases=[1] until generate() and weight processing land). The entry, header counts and the verification_history record are what the tool wrote. The slow real-weights suite (test_whisper_adapter.py) passes 6/6, and teacher-forced logits match an independent eager HF model exactly (max abs diff 0.0 over all 51865 logits at each decoder position).
verify_models on openai/whisper-large-v3-turbo: P1 at 100% (all 242 components equivalent to HF, forward-pass logits within tolerance). This is the asymmetric case (32 encoder / 4 decoder layers) with the 128-mel-bin front end of the large-v3 family. The entry, header counts and the verification_history record are what the tool wrote.
dipo101
force-pushed
the
whisper-adapter
branch
from
October 23, 2026 23:10
9672083 to
26b528e
Compare
jlarson4
reviewed
Oct 26, 2026
| ) | ||
| decoder_input_ids = kwargs.get("decoder_input_ids") | ||
| if isinstance(decoder_input_ids, torch.Tensor) and decoder_input_ids.ndim == 1: | ||
| kwargs["decoder_input_ids"] = decoder_input_ids.unsqueeze(0) |
Collaborator
There was a problem hiding this comment.
A 1-D decoder_input is promoted to a batch of one, not to the batch of features. With several clips, the forward returns a single row of logits that matches none of them, and a [1, T] prompt does the same. Can you line the decoder batch up with the features' batch (or reject a mismatch), and add a multi-clip case to the tests?
Contributor
Author
There was a problem hiding this comment.
Good catch thanks. A single decoder row is now shared across every clip, and any other mismatch between decoder_input/labels and the features' batch raises a ValueError.
Also added multi-clip cases to test_whisper_tiny.py (shared prompt, shared labels, rejected mismatch) and a two-clip parity check against HF in the slow whisper-tiny test. Fixed in 74fad08.
| """Set the audio seq2seq flags on top of the BART-family setup.""" | ||
| super().__init__(cfg) | ||
| self.cfg.is_audio_model = True | ||
| self.cfg.is_audio_seq2seq = True |
Collaborator
There was a problem hiding this comment.
bridge.to_tokens(" Hello") gives <|startoftranscript|><|notimestamps|><|endoftext|> Hello. default_prepend_bos is still on, so an end-of-text token lands inside the decoder prompt and shifts every later position. Could you make to_tokens produce a clean decoder prompt for Whisper and pin it in a test?
Contributor
Author
There was a problem hiding this comment.
Fixed in 74fad08 also. The adapter now sets default_prepend_bos = False, since the tokenizer's <|startoftranscript|> prefix already starts the prompt. to_tokens(" Hello") now gives <|startoftranscript|><|notimestamps|> Hello, or the full <|en|><|transcribe|> prefix after set_prefix_tokens.
…coder prompt - A single decoder row (1-D or [1, T]) is now shared across every clip, and any other mismatch between decoder_input or labels and the features' batch raises. Before, HF returned one row of logits that matched none of the clips. - WhisperArchitectureAdapter sets default_prepend_bos=False. The tokenizer's <|startoftranscript|> prefix already starts the prompt, so to_tokens no longer puts <|endoftext|> inside it. Tests cover several clips sharing one prompt and one label row, a rejected mismatch, and to_tokens with a download-free WhisperTokenizer (tiny) and the real one (whisper-tiny, slow).
jlarson4
approved these changes
Oct 26, 2026
jlarson4
left a comment
jlarson4
left a comment
Collaborator
There was a problem hiding this comment.
Approved for merge! I am going to cancel the running CI, there is a bug in the CI right now due to a testing model being removed from HuggingFace. I will push that fix once it is merged into dev and re-run CI
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
WhisperArchitectureAdapter, soWhisperForConditionalGenerationcheckpoints load viaboot_transformerswith hooks on both the audio encoder and the text decoder. This covers tiny through large-v3, large-v3-turbo and distil-whisper.conv1/conv2plus sinusoidalembed_positions, the unembed isproj_out, and asymmetric stacks follow the decoder.k_projhas no bias; this is documented and pinned in tests.AUDIO_SEQ2SEQ_ARCHITECTURESset (loads viaAutoModelForSpeechSeq2Seq);cfg.is_audio_seq2seqflag that keeps the tokenizer and loadsWhisperProcessor;audio_input_name="input_features", withdecoder_inputrequired.openai/whisper-tinyandopenai/whisper-large-v3-turbo(32/4 layers) are verified at P1 = 100%.Part of #1938.
generate()and weight processing follow in separate PRs; until then they raise clear errors.Type of change
Checklist: