Diogo Almeida - Deep Learning: Modular in Theory, Inflexible in Practice - TWiML Talk #8 — The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) 播客文字稿

Diogo Almeida introduces himself and his background in math olympiads, Kaggle, and medical deep learning.

Diogo works at a medical deep learning startup focused on fast, accurate, safe medical diagnoses, and previously broke a 13-year losing streak for the Philippines at the International Math Olympiad.
He won a Kaggle competition in 2013 called the Cause-Effect Pairs Challenge, where data consisted of pairs of sequences rather than tabular data, and the task was to predict causal direction between variables like altitude and temperature.
The challenge's core idea was to figure out how correlation implies causation, which is useful because observational data is abundant while controlled studies are hard to run.

Diogo explains his winning Kaggle methodology based on massive automated feature engineering.

Traditional statistical causality tests scored around 0.6 AUC as single features, while top competitors used tens to hundreds of features, but Diogo generated 50,000 features by trying every combination of machine learning algorithm and fit metric.
His rationale was that if X causes Y, then Y is a function of X, so he used the entire field of curve fitting (machine learning) to approximate functions rather than hand-engineering statistical tests.
He trained a boosted decision tree on a 50-core machine for a week, and despite nearly being overtaken on the validation leaderboard, he won because competitors overfit with hundreds of submissions.

Diogo describes how the 50,000 features were generated, including handling categorical data.

Features grew exponentially from nested loops over every combination of metric, machine learning algorithm, and symmetric feature augmentation.
Because some input was categorical, he converted between categorical and numerical data via clustering, binning, and PCA projection to the first principal component.
He also applied a feature selection algorithm to reduce the feature set, which helped performance slightly but was not ultimately important.