SPEC logo SPEC: Explainable Comparison and Alignment of Feature Embeddings

Mohammad Jalali1, Bahar Dibaei Nia2, Farzan Farnia1,
1The Chinese University of Hong Kong, 2Sharif University of Technology
Accepted at ICML 2025
OpenReview Paper arXiv Github Code Colab Quick Start (Colab) BibTex

SPEC compares and aligns feature embeddings by analyzing the eigenspectrum of differential kernel.

Abstract

While several feature embedding models have been developed in the literature, comparisons of these embeddings have largely focused on their numerical performance in classification-related downstream applications. However, an interpretable comparison of different embeddings requires identifying and analyzing mismatches between sample groups clustered within the embedding spaces.

In this work, we propose the Spectral Pairwise-Embedding Comparison (SPEC) framework to compare embeddings and identify their differences in clustering a reference dataset. Our approach examines the kernel matrices derived from two embeddings and leverages the eigendecomposition of the difference kernel matrix to detect sample clusters that are captured differently by the two embeddings.

We present a scalable implementation of this kernel-based approach, with computational complexity that grows linearly with the sample size. Furthermore, we introduce an optimization problem using this framework to align two embeddings, ensuring that clusters identified in one embedding are also captured in the other model. We provide numerical results demonstrating the SPEC's application to compare and align embeddings on large-scale datasets such as ImageNet and MS-COCO.

Overview of SPEC: Comparison and Alignment of Embeddings

SPEC performs an eigendecomposition of the difference of kernel matrices following the two compared embeddings on a given reference dataset. SPEC-align optimizes the alignment of two embeddings by minimizing the SPEC-diff.

Text Embedding Comparison using SPEC

Image Embedding Comparison using SPEC

Alignment of Embeddings using SPEC

Comparing similarity ranking for SPEC clusters

BibTeX

To cite this work, please use the following BibTeX entries:

SPEC framework for pairwise comparison of Embeddings:

@inproceedings{
      jalali2025spec,
      title = {Towards an Explainable Comparison and Alignment of Feature Embeddings},
      author = {Mohammad Jalali and Bahar Dibaei Nia and Farzan Farnia},
      booktitle = {Forty-second International Conference on MachineLearning},
      year = {2025},
      url = {https://openreview.net/forum?id=Doi0G4UNgt}
}

FINC framework for pairwise comparison of generative models:

@InProceedings{zhang2025finc,
      author = {Zhang, Jingwei and Jalali, Mohammad and Li, Cheuk Ting and Farnia, Farzan},
      title = {Unveiling Differences in Generative Models: A Scalable Differential Clustering Approach},
      booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
      month = {June},
      year = {2025},
      pages = {8269-8278}
}