Why Does Alignment Work?
I was recently curious about the alignment mechanism in those Vision-Language Models (VLMs). We know some VLMs are not inherently multi-modal by design: they glue a pretrained vision encoder to a text-only LLM, usually through a linear projection layer.1 Then joint training is introduced, probably through some multi-modal task. The practice, which also matches our intuition, works pretty well. And I’ve heard that such Frankenstein models can infer unseen characters from a pretrained vision encoder.2 However, I was still puzzled by why such joint training does not disrupt the learned representations and instead aligns different modalities pretty well. This alignment mechanism is related to much broader questions, such as why fine-tuning (in many cases) does not break the learned general knowledge, even with full fine-tuning rather than LoRA. But here let’s focus on the simple two-model alignment problem. Let’s say one is a vision model $f_v$, and the other is a language model $f_l$: ...