ISL Colloquium
The ISL Colloquium convenes in person again (on Thursdays at 4pm PT in Packard 101) as of Fall 2021. For those unable to attend, talks are also streamed via Zoom. To avoid "Zoom-bombing", we ask attendees to input their email address here to receive the Zoom meeting details via email.

← List all talks ...

Interpolation Phase Transition in Neural Networks: Memorization and Generalization

Joe Zhong – Postdoc, Stanford

Thu, 23-Sep-2021 / 4:30pm / Packard 101



A mystery of modern neural networks is their surprising generalization power in overparametrized regime: they comprise so many parameters that they can interpolate the training set, even if actual labels are replaced by purely random ones; despite this, they achieve good prediction error on unseen data.

To demystify the above phenomena, we focus on two-layer neural networks in the neural tangent (NT) regime. Under a simple data model where n inputs are d-dimensional isotropic vectors and there are N hidden neurons, we show that as soon as Nd >> n, the minimum eigenvalue of the empirical NT kernel is bounded away from zero, and therefore the network can exactly interpolate arbitrary labels.

Next, we study the generalization error of NT ridge regression (including min-$ell_2$ norm interpolation). We show that in the same overparametrization regime Nd >> n, in terms of generalization errors, NT ridge regression is well approximated by kernel ridge regression (infinite-width kernel), which is in further we approximated by polynomial ridge regression. A surprising phenomenon is a “self-induced” regularization due to the high-degree components of the activation function.

Link to the ArXiv paper:


Joe Zhong is currently a postdoc at Stanford University, advised by Prof. Andrea Montanari and Prof. David Donoho. His research interest includes statistics, optimization, and deep learning. Prior to this, Joe Zhong obtained his PhD in 2019 from Princeton University, where he was advised by Prof. Jianqing Fan, and a B.S. in mathematics from Peking University in 2014.