Supplemental Materials Menu
Deep learning networks have revolutionized the creation of artificial intelligence and have also led to new insights about neural processing in the brain.
Overview
Research on learning in deep networks has led to impressive performance by machines on tasks such as recognition in many domains. Theoretical studies such as those in this unit help us to better understand the behavior of these networks, why they perform so well, what are their limitations, and what fundamental questions remain to be answered.
Videos
Terrence Sejnowski: The deep learning revolution
Terrence Sejnowski reflects on the historical evolution of deep learning in Artificial Intelligence, from perceptrons to deep neural networks that play Go, detect thermal updrafts, control a social robot, and analyze complex neural data using methods that are revolutionizing neuroscience.
Tomaso Poggio: Dynamics and generalization in deep neural networks
The reasons for why deep learning works well for tasks such as recognition remain a mystery. Tomaso Poggio first uses approximation theory to formalize when and why deep networks are better than shallow networks, then draws on optimization theory to determine where complexity control lies in deep networks, and finally considers classical and modern regimes in learning theory and provides new foundations.
Max Tegmark: Connections between physics and deep learning
The study of deep learning lies at the intersection between AI and machine learning, physics, and neuroscience. Max Tegmark explores connections between physics and deep learning that can yield important insights about the theory and behavior of deep neural networks, such as their expressibility, efficiency, learnability, and robustness.
Haim Sompolinsky: Neural representations of categories
Haim Sompolinsky introduces neural representations of object manifolds, the problem of manifold classification, and properties of manifold geometry. He then describes applications to visual object manifolds in deep convolutional neural networks, applications to neural data, and few shot learning of objects.
Lorenzo Rosasco: The quest for provably efficient ML algorithms
Lorenzo Rosasco first introduces statistical learning from a theoretical perspective, and then describes the construction of learning algorithms that are provably efficient so that they can achieve good results with a minimal amount of computation. Finally, he presents empirical results for the case study of large-scale kernel methods.
Constantinos Daskalakis: Learning from biased data
Constantinos Daskalakis illustrates common ways that datasets constructed for training neural networks can be biased, such as censoring and truncation of data samples, giving rise to biased models. Building connections to high-dimensional probability, harmonic analysis, and optimization can lead to computationally and statistically efficient methods for solving statistical learning tasks with truncated data in a way that reduces bias.
Further Study
Online Resources
Additional information about the speakers’ research and publications can be found at these websites:
- Terrence Sejnowski (Salk Institute for Biological Studies)
- Tomaso Poggio (MIT)
- Max Tegmark (MIT)
- Haim Sompolinsky (Harvard, Hebrew University of Jerusalem)
- Lorenzo Rosasco (University of Genoa, IIT)
- Constantinos Daskalakis (MIT)
Readings
Banburski, A., Liao, Q., Miranda, B., Rosasco, L., De La Torre, F., Hidary, J., Poggio, T. (2018) Theory III: Dynamics and generalization in deep networks, axXiv:1903.04991
Chung, S., Cohen U., Sompolinsky, H., Lee, D. D. (2018) Learning data manifolds with a cutting plane method, Neural Computation, 30(10), 2593-2615
Cohen, U., Chung, S., Lee, D. D., Sompolinski, H. (2020) Separability and geometry of object manifolds in deep neural networks, Nature Communications, 11, 746
Daskalakis, C., Rohatgi, D., Zampetakis, M. (2020) Truncated linear regression in high dimensions, 34th Conference on Neural Information Processing Systems, arXiv:2007.145392007.14539
Poggio, T. A., Anselmi, F (2016) Visual cortex and deep networks: Learning invariant representations, The MIT Press, Cambridge
Sejnowski, T. J. (2018) The deep learning revolution: Artificial intelligence meets human intelligence, The MIT Press, Cambridge
Tegmark, M. (2017) Life 3.0: Being human in the age of artificial intelligence, Alfred A Knopf, New York