Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation

What came before

LoRS relaxes the rigid one-to-one pairing assumption at the *similarity-label* level via low-rank factorization of a similarity matrix, but still aligns image and text representations with full-dimensional Euclidean objectives. CovMatch matches cross-covariance between real and synthetic features uniformly across all feature directions, with a trainable text encoder, but does not distinguish which directions actually carry shared cross-modal information from which do not.

The problem

The paper diagnoses that real image-text correlation is rank-deficient: shared semantics concentrate in a low-dimensional "range" (the top directions of the image-text cross-covariance, found via SVD), while the remaining, larger residual subspace carries mostly weakly-correlated, modality-private variation. Enforcing uniform full-dimensional Euclidean alignment (as LoRS and CovMatch do) either wastes alignment capacity on the noisy residual directions or, when capacity is scarce under heavy compression (as few as 100 pairs), suppresses complementary modality-private information there entirely. Euclidean geometry also has no natural inductive bias for the hierarchical (coarse-scene-to-fine-attribute) structure of image-text semantics.

The idea

Lift image and text representations to hyperbolic space, whose geometry naturally accommodates hierarchical semantic structure, and split the alignment objective by rank: explicitly identify a low-dimensional "range" subspace via SVD of the real cross-covariance, enforce strong geodesic alignment only within that shared range, and separately regularize (rather than force-align) the orthogonal residual subspace so it can retain modality-private diversity instead of being over-compressed toward cross-modal agreement.

How it works

Synthetic image pixels and continuous text-token embeddings (with an attention mask, no discrete tokens) are optimized; a pretrained NFNet image encoder and BERT text encoder with projection heads are held frozen throughout distillation (only the synthetic parameters are trained), following an alternating procedure (Algorithm 1: 50 synthetic-update steps per 1 real-data model-update step, up to 200 iterations, real-batch features computed under stop-gradient) similar in spirit to CovMatch's fixed-per-step scheme. Projected features are lifted onto a Lorentz-model hyperboloid via an exponential map with a learnable curvature-scale factor $s$ (Eq. 3-4); pairwise geodesic distance is $d^c(u,v)=\frac1{\sqrt c}\,\text{arcosh}(-c\langle u,v\rangle_{\mathcal L})$ (Eq. 5), and a hyperbolic InfoNCE contrastive loss on synthetic pairs uses $-d^c(h^v_i,h^t_j)/\tau$ as logits (Eq. 6-7). Rank-aware capacity control (Eq. 9-11): the real batch's tangent-space cross-covariance $C_{real}$ is SVD-factorized ($C_{real}=U\Sigma V^\top$), and an adaptive rank $k$ is chosen as the smallest $k'$ whose top-$k'$ singular-value energy covers a threshold fraction $\rho=0.95$ of total energy, splitting features into a $k$-dimensional range and an orthogonal-complement residual. The range loss (Eq. 21) combines Sinkhorn-matched relevance distributions with an energy regularizer penalizing synthetic range coupling falling below the real range coupling; the residual loss (Eq. 24) matches residual relevance distributions but adds a compression regularizer penalizing residual energy exceeding range energy. The total objective (Eq. 25) is $\mathcal L_{total}=\mathcal L_{hitc}+\lambda_{range}\mathcal L_{range}+\lambda_{residual}\mathcal L_{residual}$ ($\lambda_{range}=0.8$, $\lambda_{residual}=0.4$).

Evidence

NFNet+BERT protocol; Table 4 reports averaged IR and TR (mean over K=1,5,10), not per-K breakdowns. Flickr30K@100: RAHA IR/TR/Mean 18.7/22.7/20.7, vs their reproduction of CovMatch 20.9/24.6/22.8 - RAHA trails at this smallest budget. @200: RAHA 23.5/27.9/25.7 vs CovMatch 20.2/23.7/22.0 - RAHA ahead. @500: RAHA 30.0/35.9/32.9 vs CovMatch 26.3/31.5/28.9 - RAHA ahead by a larger margin. COCO@100: RAHA 6.6/7.7/7.2 vs CovMatch 6.5/7.4/7.0 (near tie). @200: RAHA 9.3/11.0/10.2 vs CovMatch 7.4/9.2/8.3. @500: RAHA 12.6/14.9/13.7 vs CovMatch's own table row 9.9/8.3/11.2 (the paper's own printed mean of 11.2 does not arithmetically match the average of 9.9 and 8.3, an inconsistency in the source table that could not be resolved from the text). At 1000 pairs (their appendix Table A6): Flickr30K mean RAHA 38.0 vs CovMatch 28.4; COCO mean RAHA 18.6 vs CovMatch 14.2. A reproduction note (their Appendix Table A4) states published CovMatch numbers often exceed the authors' own reproduction by 7-12 points, and RAHA is reported as outperforming CovMatch under the reproduced protocol at 200/500 pairs. Cross-architecture transfer (Table 5, Flickr8K, mean recall on NF-ResNet/NF-RegNet/ViT-B): RAHA improves from 9.9 (N=100) to 12.7 (N=500) while CovMatch stays roughly flat (9.6 to 8.7). Cost: the paper states that at batch size 1 RAHA and CovMatch are similar (~25s/iteration), but the gap grows with batch size due to the SVD/lifting overhead - at batch size 64, RAHA takes ~400s versus CovMatch's ~55s per iteration, i.e. RAHA is substantially slower at scale. The key ablation (Figure 2, Flickr8K N=100) is additive: hyperbolic InfoNCE alone gives ~18 mean recall; adding range matching raises it to ~19; adding residual regularization on top reaches ~20 - both the range and residual terms contribute, and residual matching without range anchoring is reported as weak on its own.

Limitations

The paper states performance may degrade under domain-shifted or noisy captions, and that datasets with weak semantic hierarchy may see limited gains from the hyperbolic geometry (their Appendix A5). RAHA underperforms CovMatch at the smallest budget tested (100 pairs, both Flickr30K and about-tied on COCO), so the rank-aware hyperbolic approach is not uniformly better under the most extreme compression. The per-iteration cost is substantially higher than CovMatch's at realistic batch sizes (~400s vs ~55s at batch 64) due to the SVD and hyperbolic-lifting overhead, a cost not weighed against the accuracy gain in the paper. The method inherits CovMatch's dependence on already- pretrained encoders and, like the rest of the group, is bounded by the expressivity of the teacher encoder used.