Mitra, PP (May 2021) Fitting elephants in modern machine learning by statistically consistent interpolation. Nature Machine Intelligence, 3 (5). pp. 378-386. ISSN 2522-5839
Abstract
Textbook wisdom advocates for smooth function fits and implies that interpolation of noisy data should lead to poor generalization. A related heuristic is that fitting parameters should be fewer than measurements (Occam’s razor). Surprisingly, contemporary machine learning approaches, such as deep nets, generalize well, despite interpolating noisy data. This may be understood via statistically consistent interpolation (SCI), that is, data interpolation techniques that generalize optimally for big data. Here, we elucidate SCI using the weighted interpolating nearest neighbours algorithm, which adds singular weight functions to k nearest neighbours. This shows that data interpolation can be a valid machine learning strategy for big data. SCI clarifies the relation between two ways of modelling natural phenomena: the rationalist approach (strong priors) of theoretical physics with few parameters, and the empiricist (weak priors) approach of modern machine learning with more parameters than data. SCI shows that the purely empirical approach can successfully predict. However, data interpolation does not provide theoretical insights, and the training data requirements may be prohibitive. Complex animal brains are between these extremes, with many parameters, but modest training data, and with prior structure encoded in species-specific mesoscale circuitry. Thus, modern machine learning provides a distinct epistemological approach that is different both from physical theories and animal brains.
| Item Type: | Paper |
|---|---|
| Subjects: | bioinformatics > computational biology > algorithms > machine learning |
| CSHL Authors: | |
| Communities: | CSHL labs > Mitra lab |
| SWORD Depositor: | CSHL Elements |
| Depositing User: | CSHL Elements |
| Date: | 1 May 2021 |
| Date Deposited: | 02 Jun 2021 15:13 |
| Last Modified: | 02 Jun 2021 15:13 |
| URI: | https://repository.cshl.edu/id/eprint/40189 |
Actions (login required)
 |
Administrator's edit/view item |