Why Does Alignment Work?

I was recently curious about the alignment mechanism in those Vision-Language Models (VLMs). We know some VLMs are not inherently multi-modal by design: they glue a pretrained vision encoder to a text-only LLM, usually through a linear projection layer.¹ Then joint training is introduced, probably through some multi-modal task. The practice, which also matches our intuition, works pretty well. And I’ve heard that such Frankenstein models can infer unseen characters from a pretrained vision encoder.² However, I was still puzzled by why such joint training does not disrupt the learned representations and instead aligns different modalities pretty well. This alignment mechanism is related to much broader questions, such as why fine-tuning (in many cases) does not break the learned general knowledge, even with full fine-tuning rather than LoRA. But here let’s focus on the simple two-model alignment problem. Let’s say one is a vision model $f_v​$, and the other is a language model $f_l​$:

$$ z_v​=f_v​(x), z_l​=f_l​(y), $$

where $z_v \in \mathbb{R}^{d_v}​$ and $z_l \mathbb{R}^{d_l}$ may live in different spaces and have different dimensionalities ($d_v \neq d_l$).

Teleological intuition

One way to make sense of why alignment works is to suppose that both spaces share an approximately invertible transform:

$$ z_l​ \approx A z_v​, $$

in which $A$ might be a linear or nonlinear transform. What really matters is that the topology/global structure is very similar (see the Platonic representation hypothesis³; there are also examples from neuroscience showing similar stuff⁴). Examples include $||cat, dog||> ||cat, sink||$ or more complicated yet fundamental relationships like the triangle inequality. Then, a cheaper solution, from the perspective of an optimizer, is not to mess up one space (or both) completely, but to just twist them a little bit so that they can communicate. More practically, you don’t even need both to be globally transformable; a shared subspace is usually enough. And then, suppose both $f_v​$ and $f_l$ are heavily overparameterized and well-trained (they have already reached basins). Gradients should usually flow to their bottleneck bridge layer rather than to the models themselves in directions that may destroy their structure. But this does not answer the question of how SGD will prioritize solutions involving such small deformations over disruptive ones.

Mechanistic understanding

Let $\theta_0$ be the initial parameter set of both $f_v​$ and $f_l​$ and $\Delta \theta = \theta - \theta_0$ be the parameter update. Suppose we use MSE as the alignment loss: $L = {1\over 2}||r(\theta_0)||^2$. If we expand around the initial parameters, we have:⁵

$$ r(\theta_0 + \Delta \theta) \approx r_0 + J \Delta \theta. $$

With this approximation, a single step in gradient descent can be written as

$$ \frac{d\Delta \theta}{dt} = -J^\top (r_0 + J \Delta \theta). $$

Intuitively, with a “larger” $J$, we will have faster convergence to zero alignment loss. In a high-dimensional setting, this means that if we take the SVD of $J$ and let $\sigma_i$ be its $i$-th singular value, $v_i$ its singular vector, and $u_i$ its error direction, then those large $\sigma_i$s will dominate the gradient descent process:⁶

$$ u_i^\top r(t) = u_i^\top r(0) \cdot e^{-\sigma_i^2 t}, $$

and the corresponding parameter shift will be very small. Now, why is the sensitivity oriented towards the bridge part but not the models themselves? After sufficient pretraining, it has been shown that sometimes this is the case (effective fine-tuning requires a small intrinsic dimension⁷), and that’s a way to interpret why LoRA works.

At this point, I’m starting to get a rough sense of how alignment may work. And I guess those works on model merging (task arithemtics⁸, like those sakana papers⁹) may also benefit from the same principle, and future such work may also take this to improve them.