New Theory Cracks Open the Black Box of Deep Learning; Natalie Wolchover; In Quanta Magazine, also
syndicated out to copied onto Wired.com; 2017-10-09; pdf.
Teaser: A new idea called the “information bottleneck” is helping to explain the puzzling success of today’s artificial-intelligence algorithms — and might also explain how human brains learn.
tl;dr → the “information bottleneck,” an explainer; as the metaphor.
and → <quote><snip/> that a network rids noisy input data of extraneous details as if by squeezing the information through a bottleneck, retaining only the features most relevant to general concepts.</quote>
Phases of Deep Learning
- “fitting” or “memorization”
- Is shorter (than the longer phase).The network learns labels for training data.
- “compression” or “forgetting”
- Is longer (than the shorter phase).
The network observes new data, to generalize against it. The network
optimizes (“becomes good at”) generalization, as measured differential with the (new) test data.
- Alex Alemi, Staff, Google.
…quoted for color, background & verisimilitude; a booster.
- William Bialek, Princeton University.
- Kyle Cranmer, physics, New York University.
…quoted for color, background & verisimilitude; a skeptic.
- Geoffrey Hinton,
…quoted for color, background & verisimilitude; is non-committal, “It’s extremely interesting.”
- Staff, Google
- Faculty, University of Toronto
- Brenden Lake, assistant professor, psychology & data science statistics, New York University.
In which a data scientist is a statistician who performs statistics on a Macintosh computer in San Francisco; and Prof. Lake’s employer is the university system of the State of New York.
- Pankaj Mehta
- Ilya Nemenman, faculty, biophysics, Emory University.
- Fernando Pereira, staff, Google.
- David Schwab
- Andrew Saxe, staff, Harvard University.
Expertise: Artificial Intelligence, The Theory of The Science of The Study of The Neuron; a.k.a. neuroscience.
- Ravid Shwartz-Ziv, graduate student, Hebrew University, Jerusalem, IL.
Advisor: Naftali Tishby
- Naftali Tishby, Hebrew University, Jerusalem, IL.
- Noga Zaslavsky, graduate student, Emory Univerity.
Advisor: Ilya Nemenman.
- Stuart Russell, éminence grise.
- Claude Shannon, theorist.
Yes, there were papers referrenced. See notes..
In archaeological order, in Quanta Magazine…
- Clever Machines Learn How To Be Curious, 2017-09-19.
- Is Alphago Really Such A Big Deal, 2016-03-29.
- A Unified Theory Of Randomness 20160802/, 2016-08-02.
- Artificial Intelligence Aligned With Human Values Q&A With Stuart Russell, 2015-04-21.
- Deep Learning Relies On Renormalization Physicists Find, 2014-12-04.