Victor Veitch

Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms

Ziwei Su, Junyu Ren, Victor Veitch

Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes. However, surprisingly, empirical studies reveal that despite this, these "discarded" norms seem to correl…Read more