Deep Learning: What It Is & What It Can Do For You • Diogo Moitinho de Almeida • GOTO 2017

Episode Timeline

1. 0:07

   Introduction, speaker background, and why deep learning matters now

   • Diogo Moitinho de Almeida works at Google Brain and frames the talk around why the audience should care about deep learning given the current hype cycle.
   • He establishes credibility by noting he broke a 13-year losing streak for the Philippines at the International Math Olympiad and won top prize in the International Modeling competition.
   • Jeff Dean's quote that if you're not considering deep neural nets for your problems you almost certainly should be is used to motivate the audience.
2. 4:00

   What deep learning actually is: gradient descent and stacked simple functions

   • The core of deep learning is framing a problem so that what you care about (or a proxy) is differentiable, then minimizing it via gradient descent.
   • Deep learning is essentially machine learning made 'deep' by stacking many simple functions together, which yields something far more powerful than any single function.
   • The hard part is knowing that the simple algorithm will still work for very complicated, non-convex models with stacks of layers.
3. 8:17

   Survey data on AI progress and the difficulty of predicting what deep learning can do

   • A survey of AI researchers shows Angry Birds rated roughly as difficult as the World Series of Poker, and AI researcher rated significantly harder than math researcher with a roughly 50-year gap.
   • Opinions diverge wildly on AGI timelines, with some researchers predicting 10-15 years and others over a hundred years; the speaker places himself in the latter camp.
   • Andrew Ng's heuristic that any mental task a typical person can do in under one second can probably be automated is offered as a rough guide.

Key Concepts

• deep learning— The central topic of the talk, defined as stacked simple differentiable functions optimized by gradient descent.
• gradient descent— The single simple algorithm that underlies almost all recent machine learning progress.
• supervised learning— The speaker's recommended approach: label data and tell the model the correct answer directly.

Notable Quotes

• if a typical person can do a mental task with less than one second of thought we can probably automate it using AI either now or in the near future

  🔥

  — A concrete, memorable heuristic from Andrew Ng that reframes the question of what AI can automate around human reaction time.

  Jump to 8:59

• how I feel about reinforcement learning is that it can work but you don't want to rely on it working

  💡

  — Directly challenges the hype around deep reinforcement learning by warning that betting your company on it often ends in vaporware.

  Jump to 11:10

Actionable Takeaways