Analysis & theory
Optimizing Millions of Hyperparameters by Implicit Differentiation
Jonathan Lorraine, Paul Vicol, David Duvenaud
AISTATS 2020 · first public 2019-11-06 · arXiv 1911.02590
In one paragraph
Proposes an implicit-function-theorem approach to hyperparameter gradients using efficient inverse-Hessian approximations, enabling joint optimization of millions of weights and hyperparameters (including a learned data-augmentation network) at only a few times the cost of standard training — an implicit-differentiation technique later adopted by kernel-based dataset distillation methods to avoid unrolling the inner loop.
Where it sits
- Setting: Image classification
Abstract (verbatim from arXiv)
We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples. Jointly tuning weights and hyperparameters with our approach is only a few times more costly in memory and compute than standard training.
BibTeX (generated; prefer the venue's official entry)
@article{lorraine2019optimizing,
title = {Optimizing Millions of Hyperparameters by Implicit Differentiation},
author = {Jonathan Lorraine and Paul Vicol and David Duvenaud},
journal = {AISTATS 2020},
year = {2019}
}