Learning Convolutions from Scratch

Abstract

Despite their impressive performance in many applications, our theoretical understanding of why neural networks are so successful is still limited. In recent years, there has been some progress in better understanding neural networks. However, most of the theoretical work has been focused on very wide neural networks, where the analysis becomes more tractable. The dynamics and behavior of neural networks of practical sizes are not well understood.

In this talk, I will discuss a line of work on training dynamics of finite width networks and in particular how training dynamics can implicitly restrict the set of functions that can be learned and lead to better generalization. I will review some theoretical results on how different choices of initialization, optimization algorithm and architecture affect the training dynamics and discuss their connection to generalization.

Bio

Behnam Neyshabur is a Research Scientist at Google Brain. Prior to that, he was an Assistant Professor at New York University. He obtained his PhD from Toyota Technological Institute at Chicago (TTIC). His research has been focused on theoretical aspects of deep learning, optimization and generalization.