Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate

Belkin, Mikhail, Hsu, Daniel, Mitra, Partha (June 2018) Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate. arXiv. (Submitted)

[thumbnail of 1806.05161v3.pdf] PDF
1806.05161v3.pdf - Submitted Version
Available under License Creative Commons Attribution Non-commercial.

Download (454kB)


Many modern machine learning models are trained to achieve zero or near-zero training error in order to obtain near-optimal (but non-zero) test error. This phenomenon of strong generalization performance for "overfitted" / interpolated classifiers appears to be ubiquitous in high-dimensional data, having been observed in deep networks, kernel machines, boosting and random forests. Their performance is consistently robust even when the data contain large amounts of label noise. Very little theory is available to explain these observations. The vast majority of theoretical analyses of generalization allows for interpolation only when there is little or no label noise. This paper takes a step toward a theoretical foundation for interpolated classifiers by analyzing local interpolating schemes, including geometric simplicial interpolation algorithm and singularly weighted k-nearest neighbor schemes. Consistency or near-consistency is proved for these schemes in classification and regression problems. Moreover, the nearest neighbor schemes exhibit optimal rates under some standard statistical assumptions. Finally, this paper suggests a way to explain the phenomenon of adversarial examples, which are seemingly ubiquitous in modern machine learning, and also discusses some connections to kernel machines and random forests in the interpolated regime.

Item Type: Paper
Subjects: bioinformatics > computational biology > algorithms > machine learning
CSHL Authors:
Communities: CSHL labs > Mitra lab
SWORD Depositor: CSHL Elements
Depositing User: CSHL Elements
Date: 13 June 2018
Date Deposited: 13 Oct 2023 16:00
Last Modified: 28 Dec 2023 16:17
Related URLs:
URI: https://repository.cshl.edu/id/eprint/41251

Actions (login required)

Administrator's edit/view item Administrator's edit/view item
CSHL HomeAbout CSHLResearchEducationNews & FeaturesCampus & Public EventsCareersGiving