Gan, E, Reizinger, P, Bizeul, A, Juhos, A, Ibrahim, M, Klindt, D, Balestriero, R, Brendel, W, Mirzasoleiman, B (December 2025) Occam’s Razor for SSL: Memory-Efficient Parametric Instance Discrimination. Transactions on Machine Learning Research, 2025-D. ISSN 2835-8856
![[thumbnail of forumid=GFNTbsVFlP.pdf]](https://repository.cshl.edu/style/images/fileicons/application_pdf.png) |
PDF
forumid=GFNTbsVFlP.pdf - Published Version Available under License Creative Commons Attribution. Download (6MB) |
Abstract
Self-supervised learning (SSL) is the prevalent paradigm for representation learning often relying on pairwise similarity between multiple augmented views of each example. Numerous learning methods with various complexities such as gradient stopping, negative sampling, projectors, additional regularization terms, were introduced in the past years. These methods can be effective, but they require careful hyperparameter tuning, have increased computational and memory requirements and struggle with latent dimensionality collapse. Furthermore, complexities such as gradient stopping make them hard to analyse theoretically and confound the essential components of SSL. We introduce a simple parametric instance discrimination method, called Datum IndEx as its Target (DIET). DIET has a single computational branch, without explicit negative sampling, gradient stopping or other hyperparameters. We empirically demonstrate that DIET (1) can be implemented in a memory-efficient way; (2) achieves competitive performance with state-of-the-art SSL methods on small-scale datasets; and (3) is robust to hyperparameters such as batch size. We uncover tight connections to Spectral Contrastive Learning in the lazy training regime, leading to practical insights about the role of feature normalization. Compared to SimCLR or VICReg, DIET also has higher-rank embeddings on CIFAR100 and TinyImageNet, suggesting that DIET captures more latent information.
| Item Type: | Paper |
|---|---|
| Subjects: | bioinformatics bioinformatics > quantitative biology |
| CSHL Authors: | |
| Communities: | CSHL labs > Klindt lab |
| SWORD Depositor: | CSHL Elements |
| Depositing User: | CSHL Elements |
| Date: | 29 December 2025 |
| Date Deposited: | 17 Feb 2026 14:52 |
| Last Modified: | 17 Feb 2026 14:52 |
| URI: | https://repository.cshl.edu/id/eprint/42089 |
Actions (login required)
 |
Administrator's edit/view item |