“Information Bottleneck” Theory Cracks Open the Black Box of Deep Learning | Quanta Magazine

New Theory Cracks Open the Black Box of Deep Learning; Natalie Wolchover; In Quanta Magazine, also syndicated out to copied onto Wired.com; 2017-10-09; pdf.
Teaser: A new idea called the “information bottleneck” is helping to explain the puzzling success of today’s artificial-intelligence algorithms — and might also explain how human brains learn.

tl;dr → the “information bottleneck,” an explainer; as the metaphor.
and → <quote><snip/> that a network rids noisy input data of extraneous details as if by squeezing the information through a bottleneck, retaining only the features most relevant to general concepts.</quote>


Phases of Deep Learning

“fitting” or “memorization”
Is shorter (than the longer phase).The network learns labels for training data.
“compression” or “forgetting”
Is longer (than the shorter phase).
The network observes new data, to generalize against it. The network
optimizes (“becomes good at”) generalization, as measured differential with the (new) test data.


  • Alex Alemi, Staff, Google.
    …quoted for color, background & verisimilitude; a booster.
  • William Bialek, Princeton University.
  • Kyle Cranmer, physics, New York University.
    …quoted for color, background & verisimilitude; a skeptic.
  • Geoffrey Hinton,
    …quoted for color, background & verisimilitude; is non-committal, “It’s extremely interesting.”

    • Staff, Google
    • Faculty, University of Toronto
  • Brenden Lake, assistant professor, psychology & data science statistics, New York University.
    In which a data scientist is a statistician who performs statistics on a Macintosh computer in San Francisco; and Prof. Lake’s employer is the university system of the State of New York.
  • Pankaj Mehta
  • Ilya Nemenman, faculty, biophysics, Emory University.
  • Fernando Pereira, staff, Google.
  • David Schwab
  • Andrew Saxe, staff, Harvard University.
    Expertise: Artificial Intelligence, The Theory of The Science of The Study of The Neuron; a.k.a. neuroscience.
  • Ravid Shwartz-Ziv, graduate student, Hebrew University, Jerusalem, IL.
    Advisor: Naftali Tishby
  • Naftali Tishby, Hebrew University, Jerusalem, IL.
  • Noga Zaslavsky, graduate student, Emory Univerity.
    Advisor: Ilya Nemenman.


  • Stuart Russell, éminence grise.
  • Claude Shannon, theorist.


Yes, there were papers referrenced.  See notes..


In archaeological order, in Quanta Magazine

Separately noted..