An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research

Reizinger, Patrik, Balestriero, Randall, Klindt, David, Brendel, Wieland (April 2025) An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research. bioRxiv. ISSN 2692-8205 (Submitted)

PDF 10.48550.arXiv.2504.13101.pdf - Submitted Version Available under License Creative Commons Attribution. Download (727kB)

Abstract

Self-Supervised Learning (SSL) powers many current AI systems. As research interest and investment grow, the SSL design space continues to expand. The Platonic view of SSL, following the Platonic Representation Hypothesis (PRH), suggests that despite different methods and engineering approaches, all representations converge to the same Platonic ideal. However, this phenomenon lacks precise theoretical explanation. By synthesizing evidence from Identifiability Theory (IT), we show that the PRH can emerge in SSL. However, current IT cannot explain SSL's empirical success. To bridge the gap between theory and practice, we propose expanding IT into what we term Singular Identifiability Theory (SITh), a broader theoretical framework encompassing the entire SSL pipeline. SITh would allow deeper insights into the implicit data assumptions in SSL and advance the field towards learning more interpretable and generalizable representations. We highlight three critical directions for future research: 1) training dynamics and convergence properties of SSL; 2) the impact of finite samples, batch size, and data diversity; and 3) the role of inductive biases in architecture, augmentations, initialization schemes, and optimizers.

Item Type:	Paper
Subjects:	bioinformatics bioinformatics > computational biology > algorithms bioinformatics > computational biology bioinformatics > computational biology > algorithms > machine learning
CSHL Authors:	Klindt, David
Communities:	CSHL labs > Klindt lab
SWORD Depositor:	CSHL Elements
Depositing User:	CSHL Elements
Date:	17 April 2025
Date Deposited:	18 Apr 2025 18:15
Last Modified:	18 Apr 2025 18:15
Related URLs:	Publisher
URI:	https://repository.cshl.edu/id/eprint/41851

Actions (login required)

Administrator's edit/view item